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LESSONS  LEARNED  IMPLEMENTING  AIR  TRANSPORTATION 
COMPUTER  BASED  TRAINING  IN  AN  OPERATIONAL 

ENVIRONMENT 


Major  Thomas  L  Alston 
MSgt  Steven  J.  Dey 
TSgt  James  J.  Bruflodt 

Abstract 


In  November  1983  the  Military  Airlift  Command  (MAQ  commissioned  an  evaluation  of 
its  air  transportation  training  program.  The  rnulting  recommendations  focxised  on 
coordinating  the  efforts  of  on-the  -job  training  (OJT),  formal  classroom  instruction,  and 
computer  based  training  (CBT).  In  January  1985  the  1492d  Air  Transportation  Training 
Center  was  created  at  Travis  AFB,  California,  to  develop  formal  courses  and  a  Module 
Development  Center  (MDC)  to  implement  Ak  Transportation  Computer  Based  Training 
(ATCBT)  at  56  sites  around  the  world.  During  the  6  years  of  its  operation,  the  CBT 
inventory  grew  to  151  lessons.  In  this  paper  the  authors  will  share  the  lessons  learned  in 
implementing  the  original  computer  bas^  training  program.  The  areas  to  be  addressed  are 

a  system  development  -  The  first  generation  ATCBT  system  operated  on  an  8088 
microcomputer  with  limited  capabilities  and  evolved  to  the  current^r  u^  80286  computer 
system.  ^  80386  microcomputer  system  is  planned  for  implementation  in  1991/^  to 
incorporate  system  enhancements. 

b.  lesson  enhancements  -  Initial  lessons  were  formatted  in  low  end,  visual  graphics 
(CCA)  with  one  way,  sequenced  page  Sipping  lessona  As  lessons  were  critiqued  by  the 
field  and  updated  by  the  staff,  improvements  were  incorporated. 

c.  lesson  developer  training  •  individuals  assigned  to  the  MDC  are  air  transportation 
specialists  with  varying  levels  of  computer  literacy.  Maximizing  productivity  of  personnel 
assigned  for  only  3  years  presents  a  veritable  challenge. 

d.  maximizing  staff  productivity  -  Integrating  the  teaching  staff  of  the  formal  school 
into  the  ATCBT  review  process  speeded  up  lesson  review. 
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LESSONS  LEARNED 
IMPLEMENTING  AIR 
TRANSPORTATION  COMPUTER 
BASED  TRAINING  I  N  AN 
OPERATIONAL  ENVIRONMENT 

Major  Thomas  L.  Alston 

MSgt  Steven  J.  D^ 

TSgt  James  J.  Bruflodt 

latfoductjon 

“More  interesting,”  "...held  my  attention,” 
"...an  excellent  training  media."  These  are 
but  a  few  of  the  student  feedback 
comments  we  frequently  receive  about  our 
Air  Transportation  Computer  Based 
Training  (ATCBT)  system.  Perhaps  you 
are  involv^  in  a  computer  based  training 
program  or  have  heard  CBT  is  the  wave  of 
the  future  and  would  like  to  purchase  a 
system  to  take  advantage  of  its  training 
capabilities  and  enhance  your  work  force. 
In  this  paper,  we  will  share  the  lessons 
learned  over  she  years  in  implementing  our 
CBT  program  and  provide  some 
recommendations  for  preventing  or 
lessening  the  impact  of  si^ar  problems. 
Althou^  this  paper  has  been  a  group 
effort  of  three  au^ors,  it’s  an  aggregation 
of  the  experiences  of  each  member  of  the 
ATCBT  team,  past  and  present  The 
lessons  learned  will  have  their  greatest 
impact  on  other  Air  Force  career  Gelds  and 
sister  services  if  they  are  currently  using  or 
contemplating  the  implementation  of 
computer  ba^  training.  Civilian  training 
programs  mag  also  derive  beneGts  from 
this  paper.  The  focus  of  our  experiences 
will  be  on  four  distinct  areas:  system 
development,  lesson  enhancements,  lesson 
developer  training,  and  maximizing  staff 
productivity.  To  better  understand  the 
context  of  our  experiences,  we  will  first 
provide  you  with  a  brief  history  of  the 
Gictors  which  prompted  our  system’s 
development 

Supervisors  and  managers  are  always 
looking  for  more  interesting  and 
challenging  training  methods  to  increase 
worker  p^uctivity  and  Qexibility.  In  the 
Air  Force,  we  must  continuously  train  a 
young,  inexperienced  work  force  that  is 
mobile,  rotating  through  different  jobs  and 
between  numerous  locations.  Prior  to  19&5, 
the  air  transportation  career  Geld  relied 
heavily  on  correspondence  courses,  on- 


the-job  training  (OJT),  and  tape-slide 
presentations  subject  matter  to  train  its 
work  force.  DissatisGed  with  the  passive 
nature  of  much  of  its  training  i»x)gram8^ 
the  Military  Airlift  Command  (MAQ 
commissioned  an  evaluation  of  its  entire  air 
transportation  training  program  in 
November  19$3.  The  panel  recommended 
the  continuation  of  the  OJT  and 
correspondence  upgrade  programs,  the 
incorporation  of  formal  intermediate  and 
advanced  air  transportation  courses^  and  an 
interactive  computer  based  training 
program  to  provide  a  well-rounded  and 
comprehensive  training  program 
throughout  the  air  transportation 
specialist’s  career.  The  1492d  Air 
Transportation  Training  Center  was 
establ^ed  at  Travis  Air  Force  Base, 
California,  in  January  19S5  to  implement  a 
coordinate  training  program  and  act  as  a 
focal  point  for  training  issues.  The  training 
center  developed  thr^  intermediate  level 
courses  for  air  cargo  and  air  passenger 
specialists,  an  advanced  Air  Transportation 
Managers’  course,  and  the  Module 
Development  Center  (MDQ.  The  charter 
of  the  MDC  is  to  manage  MACs 
worldwide  Air  Transportation  Computer 
Based  Training  (ATCBT)  program  and 
develop  a  wortog  partner^p  wifo  the  Air 
Force  Reserve  and  Air  National  Guard 
and  their  Air  Reserve  Component 
Computer  Based  Training  (ARCCBT) 
program  which  shares  the  ATCBT  lesson 
inventoiy.  Since  its  implementation  in  19$5, 
the  ATCBT  system  expanded  to  56 
locations  worldwide,  providing  151  lessons 
to  a  training  population  in  excess  of  12,000 
personnel  'The  MDC  provides  system 
support  to  each  licensed  site,  peiforms 
lesson  reviews,  incorporates  necessary 
changes  and  upgrades  to  both  lessons  and 
the  system  operating  environment, 
processes  and  analyzes  student  data  and 
system  usage  from  each  site,  acts  as  a  focal 
point  for  review  of  new  contractor- 
developed  lessons,  develops  lessons  and 
graphics  on  a  variety  of  air  transportation 
operations,  and  provides  training 
workshops  for  Geld  training  managers:  The 
stsiff  assigned  to  the  MDC  has  varied  Gom 
six  to  nine  personnel  Air  transportation 
specialists  comprise  the  staff  and  come  to 
the  job  with  a  variety  of  computer 
experience,  Gom  computer  illiterate,  to  PC 
enthusiasts,  to  some  with  formal  college 
computer  instruction.  During  its  six  years  of 
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operation,  the  ATC3T  program  ha« 
undergone  numerous  enhmoements  as 
personnel  became  experienced  and 
comments  were  received  from  Eeld  sites 
identifying  problems  and  deficiencieSL 

System  Development 

An  ideal  CST  system  wot'ld  consist 
equipment  at  the  forefiront  of  technology  in 
both  hardware  and  software,  with  a 
complete  lesson  package  requiring  no 
changes  or  upgrades.  The  Air  Force, 
howem,  is  characterized  fay  constant 
changes  in  equipment,  job  proc^ures,  and 
technical  references  resulting  in  a  never- 
ending  series  of  changes  to  student  lessons. 
Since  a  complete  package  ot  lessons  was 
not  available,  the  Training  Division  of 
Headquarters  Military  Airlift  Command 
(HQ  MAC)  made  a  conscious  decision  to 
contract  an  initial  set  of  lessons  and  add 
lessons  in  increments.  As  the  inventory 
grew  past  100  lessons,  the  storage  capacify 
ol  the  initial  computer  system  was 

The  initial  ATCBT  qretem  operated  on  an 
8008  microcomputer  equipped  with  a  single 
20  megabyte  hard  drive  containing  both 
the  ^stem  and  lesson  inventory.  Lesson 
pictures  were  ia  Color/Graphics  Adaptor 
(COA)  resolution  graphics  (low-end 
graphics  capability).  This  system  was 
implemented  at  Qw  locations  for  testing.  A 
contractor  was  selected  to  produce  the 
initial  lessons  which  eocplained  the  specialfy 
training  standards  (STS)  or  tasks  of  the 
career  Seld  (Figure  1).  Due  to  an  ever 
growing  lesson  inventory,  MAC  quickfy 
decided  to  upgrade  to  an  802S6 
microcomputer  wi&  two  20  megabyte  hard 
drives  one  drive  for  the  operating  system, 
data  management  programs,  and  40 
lessons;  and  the  second  drive  for  the 
remaining  lesson  inventory.  Each  major 
unit  workcenter  received  a  dedicated 
ATCBT  computer.  The  initial  ATCBT 
equipment  setup  expense  was  $3,130  for  a 
single  stand  alone  training  site  and  $4,198 
for  each  central  training  manager's  site 
(Figure  2).  Additional  software  expenses  of 
$3M  were  required  for  each  stand  alone 
qrstem  and  $450  for  each  training  manager 
site  (Figure  3). 

Part  of  your  software  purchase  will  include 
your  C^T  system  environment  and  a 
run-time  module  to  execute  lessona  The 
decision  to  develop  yovir  own  software  or 


purchase  an  off-the-shelf  system  is  an 
important  one.  If  you  design  your  own 
environment,  the  software  bemmes  the 
properfy  of  your  organization  and  can  be 
copied  smd  distribute  to  ai^  new  training 
sites  that  come  on  line.  Contractor- 
developed  and  c^-the-shelf  software 
environmmts  are  copyrighted,  and  you  will 
need  to  purchase  identical  software 
packages  for  each  training  site.  This  can  be 
expensive  and  can  limit  your  capabilities 
since  no  modificetions  can  be  made  to 
copyrighted  software.  Our  corresponding 
ATCBT  software,  referred  to  as  CDSX 
and  Control,  were  contractor-developed 
and  require  a  $350  investment  per  site 
(Figure  3). 

The  CBT  system  was  reevaluated  in  1989, 
and  the  training  center  staff  determined, 
through  management  data  and  student 
feedback,  that  lesson  quality  needed 
improvement  to  increase  training 
effectiveness  and  student  usage.  In 
response  to  this;  the  decision  was  made  to 
improve  text  readabilify,  standardize  screen 
faces,  upgrade  graphic  to  the  Enhanced 
Graphics  Adaptor  (EGA)  system  (mid¬ 
level  graphics  capabilify),  and  incorporate 
student  control  features  The  system 
coDvenioD  to  EGA  resulted  in  problems 
with  file  size  in  relation  to  available  disk 
space,  graphics  and  text  screen  redesign, 
and  ad^tionsd  equipment  costs.  Of  the  138 
available  lessons,  emfy  66  could  be  stored 
for  student  use.  Lessons  were  prioritized  by 
trainee  usage  and  importance,  converted 
with  minimal  correction,  and  redistributed 
to  the  field  in  Mtuch  1990  for  use. 

To  reload  all  138  converted  lessons  and  an 
additional  13  new  lessons  developed  by  a 
contractor,  replacement  of  the  two  20 
megabyte  bard  drives  was  needed.  A  study 
detemuned  a  single  340  megabyte  hard 
drive  would  adequatefy  hold  ^  developed 
lessons  and  accommodate  the  projected 
new  lessons  under  developnient  In 
addition,  the  system  would  adequately 
support  the  more  extensive  ARi'XTBT  data 
management  requirements  for  our  Air 
Reserve  Component  counteiparts.  The 
cost  estimate  to  procure  164  of  these  340 
megabyte  hard  drive  units  was  $286,000 
horn  a  sole  source  distributor  and  would 
not  have  guaranteed  compatibility  with  our 
current  qrstem  which  was  at  the  end  of  its 
5-year  life  cyclei  A  further  comparison  was 
conducted  for  a  complete  hard  drive 
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replacement  based  on  negotiated  contract 
prices.  To  upgrade  to  an  90386 
microcomputer  replacement  would  cost  an 
estimated  $266,000  for  all  164  ousting 
terminal8-$20,000  less  than  reconfiguring 
the  5-year-old  equipment  HQ  MAC 
decided  to  replace  all  systems  and  upgrade 
to  a  faster,  more  efficient  system  with 
additional  memory  and  Vidro  Graphics 
Array  (VGA)  capability  (high-level 
graphics  capability)  for  future  development 

Choosing  computer  hardware  that  is  both 
the  industry  standsud  and  state-of-the-art  is 
essential  to  the  success  of  CBT.  System 
speed,  graphics  capability,  and  storage 
capacity  are  all  important  considerations 
and  should  not  be  sacrificed  because  of 
costs.  If  lessons  are  to  be  added  over  a 
period  of  time,  be  sure  the  system  has  the 
hard  disk  storage  capability  for  the  entire 
library  of  lessons  and  data  management 
functiona  Delaying  implementation  a  year 
in  order  to  purchase  the  most  efficient 
tystem  available  could  be  a  positive  cost 
trade  oS  to  maximize  student  usage  and 
author  effectiveness. 

l^esson  Enhancements 

Based  on  student  feedback  and  our 
developers’  experimentation,  we  have 
incorporated  numerous  changes  to  our 
original  CBT  desiga  Trainees  used  the 
ATCBT  student  comment  file  feature  to 
identify  lesson  problems,  inaccurate 
information,  and  p.  wide  feedback  to  the 
MDC  staff  on  possible  enhancements  to 
improve  lessons.  Although  not  all  lessons 
contain  the  enhancements  listed  below,  as 
each  is  reviewed  and  updated,  it  will  be 
modified  to  include  these  student-centered 
features: 

a.  Bookmark  -  Allows  lesson  reentry  at 
an  identified  interruption  point 

b  Page  browsing  -  Allows  students  to 
page  beck  or  forward  while  taking  a 
lesson. 

c.  Page  numbering  -  Provides  students 
and  authors  the  page  location  within 
lessons. 

d  Review  quizzes  -  Periodically  checks 
student  understanding  of  subject  matter 
and  reinforces  learning. 

e.  Centralized  Glossary  -  Provides 
shodent  access  to  terminology. 


abbreviations,  and  acroityms. 

L  Problem  solving  screen  masks  - 
Allows  simulated  form  data  entry  and 
computer  command  entry  training. 

g.  Enhanced  graphics  -  Allows  creation 
of  more  true-to-life  images^  forms,  and 
screen  masks. 

The  bookmark  feature  marks  the  point  at 
which  a  student  exits  from  a  lesson  prior  to 
actual  completion.  The  student  ib  able  to 
return  at  a  later  time  and  resume  the 
lesson  at  the  previousfy  marked  point 
Since  the  bookmark  relates  to  indiwdual 
shident  files,  the  actual  lesson  is  unaffected. 
This  feature  has  proven  to  be  beneficial 
since  masy  of  the  lessons  are  40  minutes  to 
an  hour  in  length.  Since  shidents  perform 
training  during  work  hours  at  ^heir  sections’ 
computer  terminals,  they  can  be 
interrupted  to  perform  actual  duties.  The 
booko^k  prompt  appears  on  a  selection 
menu  fiequentfy  throughout  each  lesson 
allowing  the  trainees  to  quickly  exit  to 
perform  a  task  and  then  remm  where  th^ 
left  off  (Figure  4). 

A  second  desirable  feature  on  the  selection 
menu  is  page  up,  sometimes  referred  to  as 
page  brow^g  (Figure  4).  This  enables 
students  to  return  to  a  previous  screen  for 
review  or  clarification.  When  a  lesson 
contains  a  series  of  text  screens,  students 
have  the  capability  to  go  back  and  reread 
pages  to  ensure  th^  understand  the 
material  presented.  Students  no  longer 
have  to  exit  from  i  lesson  am^completely 
restart  the  segment  in  order  to  review  a 
previously  displayed  page  of  information. 
Page  browsing  not  only  increases 
understanding,  but  gives  the  stjJent  more 
control  of  the  lesson  and  reduces  the 
potential  for  failw  e  of  lesson  exam.«t. 

The  capability  to  know  your  location 
within  a  given  lesson  is  beneficial  for 
students  and  authorsi  Our  page  numbering 
feature  provides  a  road  map  of  where  you 
are  among  the  total  screens  wtthin  the 
lesson  segment  being  taken  (Figure  4).  This 
feature  helpe  students  gauge  the  amount  of 
time  needi^  to  complete  the  segment  and 
is  an  essential  element  for  tte  student 
comment  file.  Students  can  identify  specific 
screen  numbers  which  may  contain 
erroneous  information  or  malfunctions, 
enabling  the  authors  to  quickfy  locate  and 
correct  the  deficiencies.  This  feature  also 
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.ds  the  subject  matter  specialist  (SMS)  in 
me  annual  review  of  lessons. 

To  reinforce  student  understanding,  each 
lesson  segment  contains  a  number  of 
review  quizzes.  These  quizzes  contain 
multiple  choice,  true/false,  or  short-answer/ 
fill-in-the-blank  questions  and  are 
strategical^  placed  throughout  the  lesson 
segment  The  location  of  review  quises 
are  based  on  the  length  and  complexity  of 
the  lesson  segment  being  taken.  Although 
we  have  no  hard,  fast  rules  for  quiz 
placement,  frequency,  or  numbers  of 
questions,  we  haw  found  the  best  ratio  is  a 
review  quiz  of  two  or  three  questions  for 
every  four  to  she  screens  of  information.  If 
first-try  questions  are  answered  incorreetty, 
the  student  is  informed  of  the  error  and  the 
lesson  then  enters  a  separate  review  unit 
This  unit  contains  the  corresponding  lesson 
information  and  is  present^  in  a  slightty 
different  format  The  student  is  then  given 
another  opportunity  to  answer  the 
question.  If  the  response  is  incorrect  a^in, 
the  answer  is  given  and  the  lesson  advances 
to  the  next  question.  If  answered  correetty, 
the  student  is  given  a  short  reinforcement 
and  the  review  quiz  continuea  Upon 
completion,  the  lesson  continues  to  the 
next  learning  area  The  review  quizzes  also 
prepare  students  for  the  segment  test  as 
they  have  a  40  to  <i0  percent  sampling  of 
the  questions  to  be  asked. 

Mai^  times,  student  misunderstanding  is 
due,  in  part,  to  not  knowing  job-related 
terminology.  In  our  origii^  ATCBT 
design,  each  lesson  had  an  individual 
glossary  containing  terms  or  acronyms 
unique  to  that  spe^c  lesson.  Since  many 
of  our  lessons  are  structured  to  build  on 
one  another,  tenns  had  to  be  duplicated  in 
multiple  lesson  glossaries.  This  requirement 
creat^  a  significant  workload  for  authors 
when  editing  or  updating  temosL  The 
glossary  function  was  red^gned  so  all 
lessons  access  a  central  glossary  of  terms. 
This  saves  time  when  performiag  edits  or 
updates,  and  also  contains  mai^  terms  or 
acroiqrms  not  contained  in  ATCBT  lessons. 
If  students  coore  across  an  unfamiliar  term 
or  acrooym  in  a  technical  reference  or 
during  the  course  of  duty,  they  can  request 
a  de^tion  from  the  glossary  the  neort  time 
they  take  a  lesson  (Figure  5).  Features  such 
as  those  mentioned  above  increase  the 
success  of  our  training  qrstem,  but  to  make 
it  complete  we  must  hold  the  student's 


attention  and  increase  the  interactive 
feature  of  CBT. 

Increased  use  of  screen  masks  with 
problem  solving  exercises  have  proven  a 
successful  tool  for  increasing  interaction 
aitd  learning  for  the  shxlentSL  One 
successful  type  of  screen  mask  shows 
students  the  forms  they’ll  use  on  the  job 
with  the  relative  loration  of  required 
entries.  This  type  screen  mask  is  used 
during  the  training  phase  and  allows  the 
student  to  complete  the  form  with 
information  fiom  a  given  scenaria  During 
the  testing  phase,  a  second  type  screen 
mask  displays  a  form  used  on  ^e  job  and 
requires  students  to  ag^in  make  correct 
entries  based  on  information  provided.  The 
use  of  such  screen  mask  interaction  builds 
student  confidence  in  accurately  completing 
required  documentation  and  can  serve  as  a 
media  for  providing  group  training  on  new 
or  modified  forms  and  procedures. 

A  second  student  interaction  project  under 
development  relates  to  our  available 
transportation  computer  systems.  With  the 
use  of  fixed  prompt  locations  and 
scenarios,  students  will  be  trained  on 
command  line  entries  required  to  log  on 
and  perform  computer  system  functions 
(Figure  4).  Simulated  databases  will  be 
comtructed  to  interact  with  the  scenarios, 
thus  providing  students  with  realistic 
results.  Screen  masks  of  available  meni» 
will  be  created  to  accuratety  depict  the 
actual  working  computer  tystems.  Students 
will  be  able  to  perform  cargo  and 
passenger  processing  and.  ..ma^esting 
procedures  on  the  ATCBT  similar  to  those 
of  the  actual  systema 

The  ATCBT  visual  representations  are  a 
significant  factor  in  effectiveness. 
Through  the  use  of  c  graces 

capability,  students  can  experience  true- 
to-life  eaomplea  of  job-related  activities. 
The  4  color  GOA  "stick-figure”  graphics 
we  once  used  have  been  repla^  by 
realistie  16  color  images  perfbrmmg  various 
tasks  (Figure  6).  Images  and  equipment  are 
now  life-likei  The  camouflage  unifbrms  are 
detailed  and  in  actual  ec^sn.  This  nuy 
seem  like  a  modest  improvements,  but  we 
find  students  can  better  relate  these  images 
to  the  task  being  performed  and  the  work 
environment  whm  the  effect  realism  is 
evident  Using  an  enhanced  graphics 
capability  affects  not  onty  graphics 
relation  and  colors,  but  improves  text 
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quality  and  allow  greater  use  of  screen 
maska  Designing  an  extensive  graphics 
library  can  be  time  consuming,  but  with  the 
right  equipment,  such  as  c^or  scanners, 
you  can  reduce  production  time 
considerabfy. 

A  successful  CBT  system  relies  heavi^  on 
student-centered  features  and 
enhancements  to  maximize  performance 
tasks.  Features  such  as  bookmark,  page 
numbering,  and  page  browsing  provi^ 
stxidents  with  greater  control  and  flexibility. 
Also,  an  elective  system  reinforces 
material  with  frequent  quizzes  and  repeat 
instruction  if  the  question  is  missed.  The 
ATCBT  tystem  can  respond  directty  to  an 
unfamiliar  vocabulary  mquiiy  through  the 
availability  of  an  extensive  glossary  of 
terms  and  abbreviations.  Next,  it 
encourages  student  interaction  throu^  the 
use  of  screen  masks  which  simulate  forms 
and  computer  systems  used  on  the  job.  It 
also  encourages  the  use  of  the  students' 
thought  processes  and  decision-making 
abilities  through  true-to-life  scenarios. 
Finalty,  it  contains  an  extensive  graphics 
library  to  visually  reinforce  the  material 
With  C  these  added  features,  the  one 
question  remained,  "did  the  students  in  the 
field  actually  spend  more  time  viewing  arid 
completing  flie  redeveloped  ATOT 
lessons?" 

Our  converted  EGA  system  with  66  lessons 
was  loaded  at  each  of  our  sites  in  April 
1990.  We  anatyzed  lesson  completion  data 
for  the  66  lessons  for  the  periods  of  April 
through  August  in  1909  and  1990  to 
determine  student  acceptability  of  the  new 
qrstem.  Our  study  showed  an  average 
monthly  increase  of  68  percent  in  the 
number  of  personnel  completing  the  EGA 
lessons  versus  the  CGA  lessons.  It  could  be 
argued  that  CGA  tystem  offered  138 
lessons,  thus  affecting  our  analysis. 
However,  when  comparing  lesson 
completion  data  for  all  lessons  available  on 
each  of  the  tystems,  the  upgraded  EGA 
system  still  showed  a  54  percent  increase  in 
lesson  completion  over  the  entire  CGA 
lesson  inventory.  Furthermore,  our  analysis 
revealed  there  has  been  an  average  40 
percent  increase  in  the  numbw  of 
supervisors,  managers,  and  non-unit 
trainees  completing  the  EGA  lessons 
versus  the  CGA  lessons.  Considering  an 
estimated  15  percent  decrease  in  our  career 
field  manpower,  analysis  of  available 


student  usage  data,  and  student  comments 
from  our  sites,  the  EGA  upgrade  to  our 
ATCBT  was  both  positive  ai^  beneficial 

Faced  with  a  limited  staff  and  a  seemingty 
insurmountable  workload  of  upgradfag 
lessons  with  the  new  enhancements 
discussed,  we  initiated  a  cost  studty  on  the 
feasibility  of  contracting  the  lesson 
revisions.  All  revision  items  were 
considered  and  cost  estimates  were  based 
on  reproduction  hours.  The  study  revealed 
the  costs  associated  with  lesson  revisions 
were  nearly  equal  to  the  development  of  a 
new  lesson.  Limiting  factors  to  consider 
with  contracting  seomd  party  productions 
are  their  unfamiliarity  with  the  technical 
content  or  references  in  lessons,  limited 
knowledge  of  our  lesson  standards  and 
screen  formats,  and  time  lost  due  to 
mailing  materials  between  producers  and 
reviewers.  An  alternative  to  contracting 
entire  lesson  revisions  is  to  have  a 
contractor  onty  “rekey”  a  lesson  in  the 
authoring  language-  Screen  prints  of 
required  text  rewrites  and  redesigned 
graphics  would  be  provided  fay  the  MDC 
staff.  The  cost  of  thu  approach  is  not  final, 
but  ifs  estimated  to  one-third  of  the 
initial  shxty's  coat  quote,  but  with  the 
MDC  responsible  for  a  considerable 
portioa  The  third  option  is  for  our  MDC 
staff  to  perform  the  entire  lessems  revision. 
This  would  require  us  to  have  a  highly 
qualified,  well  trained,  and  suffdentty 
staffed  pool  of  lesson  developers  which  we 
do  not  have. 

1  niwt.lr»pftr  Training 

The  ideal  staff  consists  of  the  specialists  one 
needs,  in  the  numbers  required  to 
accomplish  the  workload,  and  provided 
with  the  equipment  needed  to  mariTniyjt 
their  productivity.  Because  we  are  a 
military  organization,  our  staff  has 
remained  relatively  constant  and  our 
budget  has  been  constrained  which 
precludes  us  from  obtaining  additional 
equipment  immediatety  tLii:  would 
improve  productivity.  As  stated  earlier,  the 
MDC  h^  been  staffed  by  six  to  nine 
personnel  over  the  past  two  years  with 
varied,  but  limited,  experience  levels. 
Personnel  assigned  to  the  MDC  normally 
serve  a  thr^year  tour  as  a  lesson 
developer  and  then  return  to  their  primary 
air  transportation  joK  This  poaes  a  staff 
training  problem  as  newty  assigned 
personnel  are  often  unfamiliar  with 
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authoring  techniquea  To  complicate 
matters,  there  is  no  Department  of  Defense 
(DOD)  affiliated  school  providing 
instruction  on  our  authoring  language. 
Available  non-DOD  schools  teach  a  n^riad 
of  other  languages  and  systems  with  tittle 
correlation  to  each  other  or  our  ATCBT 
language.  Hence,  it  became  imperative  to 
devebp  an  MDC  OJT  program  to 
maximize  the  prodiwtivi^  of  the  personnel 
during  their  short  3  to  4  years  at  the  center. 

The  development  of  an  effective  OJT 
training  program  was  not  an  easy  task. 
Since  we  ^d  no  one  eaiperienced  in 
ATCBT  lesson  development  or  the 
authoring  language  used,  it  was  difficult  to 
put  together  any  type  of  training  program. 
In  the  initial  stages  of  ATCBT 
implementation,  the  staff  directed  its 
attention  at  getting  sites  up  and  running 
with  the  lessons  being  d^eloped  by  a 
contractor.  After  the  contract  eatpired  and 
as  problems  or  inaccuracies  were  identified, 
it  became  apparent  that  our  MDC  staff 
would  need  to  acquire  the  authoring  skills 
in  order  to  nmintain  the  inventory  of 
lessons.  The  ATCBT  staff  read  books  and 
articles  on  vstrious  aspects  of  computer 
operations,  taught  themselves,  and  then 
shared  the  informatiaD  and  skills  with  other 
staff  members.  Using  this  hit-and-miss 
technique,  the  staff  achieved  an  acceptable 
level  of  efficiency  to  cope  with  the 
problems  that  arose.  However,  the  staff 
soon  realized  a  formal  program  to  tram 
future  authors  was  needed. 

The  MDC  staff  set  about  organizing 
training  requirements  ranging  from  simple 
system  command  functions,  such  as  disk 
operating  system  features,  to  the  actual 
internal  operation  of  the  ATCBT  and 
authoring  environment.  Training 
requirements  were  documented  and  an 
incremental  training  plan  was  established. 
Staff  members  were  first  trained  on  basic 
computer  operations  and  equipment 
maintenance  and  troubleshooting 
procedures.  Our  second  step  was  to  train 
them  on  the  use  of  various  word 
processing,  graphics,  and  hard  disk 
management  software  packages.  Finally, 
personnel  were  trained  on  lesson 
development  using  the  authoring  language. 
The  overall  methods  of  training  were 
comprised  of  video  tape  instruction,  user 
manual  reviews,  and  actvial  hands-on 
demonstration  and  performance 


evaluation. 

Selecting  MDC  personnel  based  on 
outstan^g  performance  as  an  air 
transportation  specialist  did  not  alw^ 
prove  effective.  Sraoe  of  the  initial^ 
assigned  air  transportation  specialists  found 
working  with  computers  unchallenging  and 
preferr^  to  ^  formal  classroom 
instruction.  Hence  several  months  were 
spent  training  personnel  and  receiving 
minimal  results.  Our  onfy  alternative  was  to 
redefine  job  prerequisites.  By  making  a 
conscious  attempt  to  recruit  personnel  who 
possessed  some  programming  skills  or  had 
experience  in  computer  operations,  we 
haw  since  reduced  training  time  fay  60 
percent  Those  individuals  with  computer 
experience  reqiiire  minimal  trainer 
supervision  and  are  self  motivated  to 
quickly  learn  our  authoring  language  on 
their  owa  Along  with  an  efficient  s^  of 
lesson  develc^jers,  you  must  have  an 
experienced  st^  of  technical  reviewers  to 
ensure  lesson  aocura^. 

Manrimiamp  fStaff  Pfrriuctivity 

Producing  a  quali^  lesson  requires  the 
coordination  of  tenon  developers  and  a 
technical  review  staff.  Both  are  interrelated 
and  work  together  to  complete  the  final 
CBT  lesson  product  While  it  is  desirable 
for  authors  to  be  qualified  in  all  aspects  of 
lesson  development,  this  is  not  always 
possibleL  Some  authm  are  better  suited  to 
make  text  and  programming  entries  into 
lessons,  while  others  are  more  skilled  b 
graphics  design.  All  authors  undergo  initial 
lesson  development  b  both  an^  and  their 
capabilities  are  evaluated.  Their  specific 
skills  and  preference  determbe  if  th^  are 
assigned  as  an  author  or  graphics  specialist 
(Figure  7).  Because  of  bternal  motivation, 
their  productivi^  is  significant^  enhanced. 

As  explabed  earlier,  our  new  ATCBT 
lessons  are  reviewed  for  technical  content 
by  a  sufagect  matter  specialist  (SMS).  Prior 
to  19$9,  lessons  were  distribute  to  various 
units  or  headquarters  staff  personnel  for  a 
SMS  review.  This  process  required  the 
mabtenance  of  an  extensive  SMS  location 
list  and  required  frequent  updatbg  sbce 
military  penoonel  move  approximately 
every  thrw  years.  A  key  problem  with 
usbg  various  unit  level  SMS  personnel  is 
the  tendency  for  knowledge  of  localized 
procedures  to  conflict  with  command-level, 
established  procedures.  Due  to  the 
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availability  of  technical  instructors  at  the 
training  center  and  because  the  SMS 
review  process  was  so  time  consuming,  the 
program  of  tracking  command  SMS  was 
discontinued.  Instructors  from  the 
Intermediate  Air  Transportation  (lAT), 
Intermediate  Wartime  Contingency  (IWC), 
and  Air  Transportation  Managers'  (ATM) 
courses  were  readity  available  to  provide 
the  MDC  the  technical  eacpertise  for  lesson 
reviews.  Due  to  the  thre^year  tenure  of 
instructors,  there  is  a  constant  turnover  of 
personnel  from  various  commands  and 
theaters  of  operation  which  precludes  bias 
during  multiple  reviews.  Also,  daily 
instructor  and  student  interaction  provides 
a  valuable  source  of  information  relating  to 
the  actual  conditions  and  changes  occurring 
in  the  air  transportation  field  By  utilizing 
these  instructors,  we  have  reduced  the 
technical  review  process  by  an  estimated  65 
percent  Also  at  our  disposal  are 
experienced  personnel  locally  assigned  to 
the  60th  Aerial  Port  Squadron.  Their 
familiarity  with  the  cargo  and  passenger 
computer  tystems  is  a  ^finite  advantage, 
and  we  are  able  to  control  the  amount  of 
local  procedure  bias  during  lesson  review. 
The  results  are  tnore  techmcally  accurate, 
better  designed,  and  more  interactive 
ATCBT  lessons. 

Each  CBT  program  must  be  tailored 
around  training  needs,  subject  areas,  and 
student  capabUities.  While  there  are  no 
clear  cut  guidelines  for  CBT  tystem 
development,  we  can  offer  sugg^ons 
from  our  experience  to  make  your 
transition  to  CBT  less  painful  To 
complement  your  authoring  software,  you 
will  need  a  well-trained  staff  of  lesson 
developers  and  graphics  specialists. 
Carefulty  screen  prospective  employees 
and  tty  not  to  hire  st^  on  a  temporary 
bests  to  ensure  continuity.  Authors  should 
be  proficient  in  computer  operations  and 
language  and  have  a  better  than  average 
knowl^ge  of  English  grammar  to  prepare 
effective  and  correct  information^  texts. 
Authors  should  be  familiar  with  lesson 
content  and  have  some  job  knowledge  of 
the  areas  being  developed.  If  a  staff  of 
subject  matter  specialists  is  used,  attempt  to 
recruit  them  locally  to  avoid  "long 
distance”  coordination  with  lesson  authors. 

Conclusion 

The  success  of  any  computer  based  training 
system  is  a  blend  of  the  proper  system  and 


support  equipment,  user  friendty  lessons, 
and  a  trained  and  productive  sti^  If  one 
rushes  into  devek^ing  a  CBT  {xogram 
without  thinking  thi^gh  all  of  its 
components,  the  program  product  will 
suffer  the  ill  effects  of  impatience.  In  this 
paper,  we  have  provided  a  brief  overview 
of  our  ATCBT  program,  the  hardware  aixl 
software  proble^  we  experienced  in  its 
evolution,  actions  taken  to  solve  them,  and 
our  sugg^ons  on  how  you  can  effectivety 
approach  implementing  CBT  in  your  own 
environment  We  also  discussed  positive 
lesson  enhancements  which  can  be 
incorporated  into  your  tystem  to  make  it 
more  user  friendly  and  reinforce  learning. 
Finally,  we  considered  the  effective  use  ^ 
available  manpower  to  achieve  our  CBT 
objective  by  estaUishing  a  comprehensive 
on-the-job  training  program  and 
maximizing  the  use  of  in-house  staff  as 
subject  matter  specialists.  Not  all  problems 
can  be  identifi^  before  tystem  purchase 
and  implementation.  We  hope  this  paper 
has  provided  you  with  a  basis  on  which  to 
evaluate  your  current  CBT  program  or 
considerations  to  think  about  if  you're 
desiring  to  enter  the  exciting  world  of 
interactive  computer  learning. 

SUMMARY  OP 
LESSONS  LEARNED 

System  Development 

-  Thoroughty  evaluale~  computer 
hardware  requirements 

Predetermine  required  lesson 
inventory  and  operating  software. 

-  Select  operating  software  common  to 
the  indiistiy. 

-  Design  lessons  to  the  maximum 
capability  of  the  equipment  used  to  run 
them. 

l>B»nn  Rnbnneementii 

-  Initiaily  incorporate  student  oriented 
coDtrdlable  features  in  your  CBT. 

-  Include  frequent  review  quizzes  to 
reinforce  learning 

-  Use  screen  masks  and  true-to-life 
scenarios  to  improve  training 
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effectiveness. 

-  Design  lessons  to  react  similar  to 
actual  computer  qrstems  or  equipment 

-  Use  realistic  graphics  images  to 
enhance  understanding. 


-  Develop  an  effective  development 
staff  training  program. 

-  The  authoring  system  used  should 
have  an  effective  tutorial 

-  First  lessons  designed  should  teach 
lesson  development  and  standards. 

-  Utilize  personnel  knowledgeable  in 
CBT  development  procedures,  with 
computer  eacperience. 

-  Tty  to  avoid  frequent  personnel 
tiunovers. 


.  Organize  CBT  development  into  areas 
of  authoring,  graphics  development 
and  subject  matter  review  specialists. 

-  Have  a  readily  available  staff  of 
subject  matter  review  specialista 

-  Structure  an  effective  cycle  for  lesson 
review. 
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Figure  1.  Speciality  Training  Standard  Example 


equipment  at  central  site 


ITEM 

Z-248,  with  360K  and  20 
20  MB  Hard  Drive 
Color  Monitor 
Printer  .  LO 
Anti  Static  Mat 
Surge  Suppressor 
40  MB  Tape  Drive 
Computer  Work  Center 
Dust  Buster 
Desk  Chair 

Halon  Fire  Extinguisher 
Disk  File  Box,  Locking 


TOTAL 


COST  PER  UNIT 
MB  Dr i ves  1 , 960 

280 
300 
573 
70 
20 
395 
350 
70 
100 
50 
30 


4,198 


EQUIPMENT  AT  REMOTE  SITE 


ITEM 

Z-24a  ,  with  360K  and  20MB  Drives 

20  MB  Hard  Drive 

Color  Monitor 

Anti  Static  Mat 

Surge  Suppressor 

Computer  Work  Center 

Desk  Chair 

Halon  Fire  Extinguisher 


COST  PER  UNIT 


1 


960 

280 

300 

70 

20 

350 

100 

50 


TOTAL 


3.130 


NOTE 

The  figures  shown  reflect  the  government  purchase  prices 
for  1985  through  1989.  They  do  not  reflect  current 
prices  and  do  not  apply  to  non-government  agencies. 


FIGURE  2. 


ATCBT  EQUIPMENT  EXPENSE  AT  EACH  TRAINING  SITE 


SOFTWARE  -  CENTRAL  SITE 


ITEM 

CDSX 

Control 

Optimize 


TOTAL 


ITEM 

CDSX  Software 
Con  tro 1 


TOTAL 


The  figures  shown 
for  1985  through, 
prices  and  do  not 


FIGURE  3.  ATCBT 


COST  PER  UNIT 
150 
200 
100 


450 


SOFTWARE  -  REMOTE  SITE 

COST  PER  UNIT 
150 
200 


350 


NOTE 

reflect  the  government  purchase  prices 
1989.  They  do  not  reflect  current 
apply  to  non- government  agencies. 


SOFTWARE  EXPENSE  AT  EACH  TRAINING  SITE 


At  the  prompt,  type  R  for  reservations,  a  6-posltlon  MAC  channel  and  a 
Julian  date  frame  (3-posltion  begin,  3— position  end).  Separate  the  I  terns 
with  semicolons  (;).  This  connmand  Is  used  only  for  TDY  requests. 

For  example,  let's  make  the  reservation  for  MAJ  Morgan.  She  will  need  a 
reservation  on  MAC  channel  STLFRF  with  a  Julian  date  frame  204  to  20G. 

Type:  R ; STLFRF ; 204; 206  and  press  the  transmit  key. 


F5 

F7 

FIO 

PgUp 

Any  Other  Key 

Glossary 

Bookmark 

Quit 

Previous  Page 

Continue 

«««BRK[ 

RESV: 

[001 

[STLFRF 

[PU 


-/- 


A  split  Reservation  Mask  will  appear  on  the  screen.  Up  to  13  flights  on 
the  MAC  channel  and  within  the  time  frame  you  requested  will  be  displayed 
under  FLIGHT  STATUS. 


F5 

F7 

FIO 

PgUp 

Any  Other  Key 

Glossary 

Bookmark 

Qui  t 

Previous  Page 

Continue 

Figure  4.  Menu  Options  and  Screen  Mask 


NAME  [ -  GRADE  - FLIGHT  STATUS - 

STS  REQ[-  PAX  CAT  MSN  «  DEP  D/T  CR  OPN  RSV  ARR  D/T 

CHANNEL [3  PRIORITY  A-NE41  2761730  B  101  000  04  0000 

TYP  TVL[  Z  SPON  SVC  B-NE41  2781B30  B  267  000  06  0600 

MONTH  [ - ,-, - /-TVL  PERF  C-NE41  2031730  B  22  000  11  0000 

MSN  SLT[T  SOURCE  D-NE41  2OB1E30  B  143  000  13  0600 

ROUT  ID[ -  RIC  E-NE4B  2001 31 B  B  90  000  17  0000 

SH/SP  [M/-  TRAN/N  AVL  F-ME41  2021 B30  B  146  000  20  0600 

-CIC  -PET  S/W/C  G-B02XX  2031000  YG  _1  000  21  0740 

MSN  NBR[ - , -  DP  D/T-L  H*462X1  2041 24B  YG  '3  000  22  0B40 

[-  EXC  BAG  I-NE4B  2601 31 B  B  IB  000  24  0000 

ORG  STN[ -  FIN  DES  J-NE41  29O1B30  B  61  000  27  0600 

FB/R  0V[ —  SVC  USE  K-NE4B  3031 31 B  B  132  000  31  1000 
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Let's  step  through  each  key  In  a  DD  Form  1387-2  to  see  where 
the  data  comes  from  so  you  will  know  where  to  go  to  verify 
its  accuracy. 


Type  in  the  word  you  want 


def ined 


..aerial  port  of  embarkation 
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Let’s  step  through  each  key  in  a  DD  Form  1387-2  to  see  where 
the  data  comes  from  so  you  will  know  where  to  go  to  verify 
its  accuracy. 


(APOE)  -  A  STATION  WHICH  SERVES  AS  AN  AUTHORIZED  PORT  TO  PROCESS  AND 
CLEAR  AIRCRAFT  AND  TRAFFIC  FOR  DEPARTURE  FROM  THE  COUNTRY  IN  WHICH 
LOCATED . 

Is  there  another  term  you  would  like  defined?  (Y/N) 


Figure  5 .  Glossary  Input  and  Response  Screens 

1  5 


Figure  7.  Module  Development  Center  Training  Cycle 


CONSroERATIONS  IN  PLANNING  FOR  EXPORTABLE 
INTERACTIVE  COURSEWARE  (ICW) 


T.  Kent  Thomas 

Exportable  training,  providing  training  to  the  trainee  when  and  where  it  is  needed,  offers  significant 
promise  to  meet  the  challenges  of  increased  training  requirements  and  decreased  training  resources.  Just 
as  today's  economic  changes  call  for  significant  shifts  in  the  way  we  do  business,  such  as  increasing  the 
use  of  exportable  training,  the  use  of  exportable  training  brings  a  ’new  set”  of  considerations  and  issues 
into  play.  The  challenge  of  implementing  an  exportable  training  program  is  further  compounded  when 
using  newer  training  technologies  such  as  interactive  courseware  (ICW).  The  author  will  present  some  of 
the  issues  addressed  by  Tactical  Air  Command  (TAC)  in  implementing  and  managing  an  exportable  ICW 
training  program  for  aircraft  maintenance  continuation  training. 

Considerations  addressed  include  purely  planning  factors,  such  as  hardware  availability,  compatibility 
and  configuration;  hardware  location  and  management;  distribution  media  and  methods;  and  courseware 
configuration  control.  Courseware  design  requirements,  trainee  and  manager  instructions  for  using  the 
courseware,  and  computer-managed  instruction  (CMI)  requirements  also  warrant  special  consideration. 

Lessons  learned  from  this  innovative  use  of  training  technology  have  widespread  application  to  others 
considering  exportable  ICW,  regardless  of  any  differences  in  training  requirements,  subject  matter  or 
target  population.  While  exportable  training  using  ICW  holds  significant  promise,  it  is  quite  different 
from  the  use  of  ICW  in  a  resident,  closely  supervised  environment.  To  capitalize  on  the  potential, 
managers  must  recognize  and  address  these  differences  -  from  conception  through  implementation. 


MR.  THOMAS  was  formerly  an  Air  Force  Education  and  Training  officer  and  a  computer  systems 
requirements  analyst.  With  a  B.A.  from  Carson-Newman  College,  he  is  now  completing  a  Masters  in 
Instructional  Technology  from  Utah  State  University.  From  1981-1985,  he  was  a  Training  Systems 
Analyst  at  HQ  ATC/TTX.  He  served  as  ATC  focal  point  for  student  management  hmctions  in  the 
Advanced  Personnel  Data  System  (APDS)  and  project  officer  for  the  Branch  Level  Training  Management 
System  (BLTMS),  ATC's  largest  operational  computer-managed  instruction  (CMI)  system.  During 
1985-86,  he  served  as  Chief,  Training  Initiatives  Branch  at  HQ  TAC/LGQT,  where  he  helped  plan 
TAC's  ICW  initiative.  He  activated  the  4400th  Maintenance  Training  Fligh'  (MTF),  TAC's  maintenance 
ICW  development  and  management  center,  assuming  command  in  1986.  The  Tactical  Air  Forces  (TAC, 
PACAF,  USAFE,  AFRes,  and  ANG)  now  have  over  30  interactive  videodisc  (IVD)  maintenance  courses 
implemented  throughout  the  world,  operating  on  approximately  300  training  systems.  Over  30  more 
courses  are  in  work,  by  both  contractors  and  4400  MTF  personnel.  The  4400  MTF  has  developed  nine 
IVD  courses  internally,  including  the  winners  of  the  Nebraska  Videodisc  Design/Production  Group's 
award  for  best  govemment/military  achievement  in  both  1990  and  1991.  He  is  now  the  Director  of 
Commercial  Courseware  Development  at  Allen  Communications,  Inc.,  Salt  Lake  City,  Utah. 


CONSIDERATIONS  IN  PLANNING 
FOR  EXPORTABLE 
INTERACTIVE  COURSEWARE 
(ICW) 

T.  Kent  Thomas 

Introduction.  ICW  can  provide  an  increased 
volume  of  training  with  a  decreased  number  of 
instructors  and  support  equipment.  The  actual 
course  content  is  presented  by  the  training 
system,  not  via  an  instructor's  lecture  or 
demonstration.  If  the  courses  are  designed 
accordingly,  the  systems  can  be  used  in  a 
"stand-alone*  mode,  requiring  little  or  no 
supervision.  The  computer  can  also  do  most  of 
the  training  management  functions  such  as 
recording  grades  or  making  assignments. 
Incredibly  large  economies  of  scale  are  possible 
using  ICW,  since  the  training  systems  are 
relatively  inexpensive,  "transparent"  to  the 
course  content,  and  courses  can  be 
inexpensively  reproduced  in  large  quantities. 
Consequently  "exporting"  ICW  to  the  students, 
instead  of  sending  the  students  to  the  school, 
has  significant  potential. 

Assuming  that  the  training  requirements  have 
been  thoroughly  analyzed  and  ICW  is  an 
appropriate  media,  several  other  factors  should 
be  considered  before  deciding  to  use  exportable 
ICW.  The  new  AFP  50-68,  Information  for 
Designers  of  Instructional  Systems,  Volume  V, 
teractive  Courseware  (ICW)  Decision  Guide, 
nu\^  provides  some  excellent  guidance  on 
considerations  for  exportable  ICW.  This 
discussion  follows  the  general  sequence  of  the 
decision  aids  in  AFP  50-68,  Vol  V,  expanding 
on  what  we  consider  the  most  crucial  factors 
based  on  our  experience,  as  briefly  described 
below. 

Tactical  Air  Command  (TAC)  is  currently 
developing  an  extensive  continuation  training 
program  for  aircraft  maintenance  technicians 
using  exportable  ICW.  The  interactive 
videodisc  (IVD)  training  systems  are  in  learning 
centers  located  in/adjacent  to  maintenance 
workcenters.  They  provide  refresher  and 
advanced  training  on  a  time-available  (i.e. 
opportune)  basis.  More  than  330  IVD  systems 
are  installed  throughout  the  world,  supporting 


over  30  courses,  primarily  for  the  F-15  and  F- 
16  aircraft.  More  than  30  additional  courses  are 
under  development. 

The  4400th  Maintenance  Training  Flight  (MTF) 
was  activated  to  provide  TAC's  "in-house"  IVD 
development  and  management  capability. 
Initially  the  4400  MTF  developed  nine  courses 
for  existing  F-15A/C  model  aircraft.  However, 
that  role  has  rapidly  changed  to  primarily 
defining  and  managing  ICW  development 
contracts,  plus  maintaining  and  updating 
courses  provided  through  the  weapons  system 
acquisition  process  or  other  contracts.  This 
approach  ensures  as  rapid  an  initial  development 
as  feasible,  responsive  courseware  updates  to 
maintain  configuration  control,  and  minimal 
contractor  dependence. 

Haviiig  determined  that  ICW  is  an  appropriate 
media  and  there  is  a  potent*  al  application  for 
exportable  ICW,  you  should  thoroughly 
research  that  potential.  ICW  and  training 
systems  are  expensive,  so  there  are  large  risks 
associated  with  this  decision  -  from  purely  a 
cost  perspective.  Let's  examine  in  detail  some 
specific  considerations  when  making  this 
decision. 

Existing  Hardware  Systems.  Obviously, 
you  should  determine  if  there  are  training 
systems  available  to  support  the  ICW.  Even 
thou-^h  microcomputers  are  becoming 
widespread,  do  not  assume  that  they  will  be 
available  to  run  t!ie  courseware.  Instead, 
conduct  an  in-depth  survey  of  the  potential 
training  sites.  You  should  be  very  familiar  with 
computer  systems,  or  you  must  get  someone 
who  is  to  help  you. 

Defining  Existing  Systems.  Determine  the 
specific  type(s)  and  number  of  machines 
available,  including  their  exact  configuration 
((operating  system,  amount  of  random  access 
memory  (RAM),  number  and  size  of  storage 
devices,  all  peripherals,  type  of  graphics  card 
and  monitor,  available  expansion  slots,  etc.)) 
For  example,  there  are  still  numerous  Z-100 
computers  in  the  Air  Force  which  are  not 
compatible  with  the  later,  more  common  Z-248 
computers  unless  they  have  been  upgraded  with 
a  Gemini  board.  Even  with  Gemini  boards, 
almost  all  Z-lOOs  are  monochrome  systems 


which  limit  their  capability  as  a  training 
pL  tform.  The  individual  Z-248s  may  also  have 
different  graphics  cards  (EGA  or  VGA).  The 
Unisys  computers  from  the  Desktop  III  contract 
can  also  have  different  graphics  cards. 

Graphics  capability  is  the  largest  single 
contributor  to  ICW  incompatibility. 

Next,  determine  how  many  vacant  expansion 
slots  are  in  each  computer,  in  case  expansion 
becomes  necessary.  Since  most  ICW,  like  most 
software,  is  distributed  on  floppy  diskettes, 
determine  what  type  of  floppy  diskette  drives 
the  systems  have?  There  are  at  least  two 
common  formats  for  both  the  S.2S  inch  and  the 
3.5  inch  diskettes.  You'll  want  to  avoid 
distributing  the  courseware  in  multiple  formats 
if  all  possible.  Do  the  systems  have  or  do  they 
need  modems  to  communicate  with  your 
computers  to  upload  or  download  lesson  or 
student  files? 

Defining  Their  Availability.  Determine  how 
the  systems  are  currently  being  used,  when, 
how  often,  for  how  long,  and  by  whom.  The 
hardware  simply  may  not  be  practical  for  use  as 
a  training  system  due  to  the  level  or  priority  of 
current  use.  For  example,  the  available  storage 
on  the  hard  disks  may  be  insufficient  for  your 
ICW.  Or,  the  potential  students  would  be 
frequently  interrupted  by  "priority"  users.  Is 
the  system  available  when  the  students  will  be, 
when  the  training  needs  to  occur,  or  is  it  in  a 
locked  office?  Consider  the  amount  of  time  the 
systems  would  be  available  for  use  by  students. 
Multiply  the  estimated  time  for  a  student  to 
complete  your  ICW  by  the  anticipated  number 
of  students  per  site.  Is  there  sufficient  time 
available  for  training  during  the  required 
period?  Don't  forget  to  consider  the  impact  of 
personnel  turnover  in  this  computation.  You 
can  also  reasonably  expect  that  computer  use  for 
their  other  intended  functions  will  increase,  so 
allow  for  some  "growth." 

Is  the  hardware  physically  located  where  the 
students  will  be,  or  will  the  students  need  to  go 
to  a  different  location  to  use  the  system?  The 
requirement  to  travel  even  a  short  distance  will 
likely  reduce  utilization  of  the  ICW.  Note  that 
one  of  the  strengths  of  exportable  ICW  is  the 
potential  use  right  in  the  workcenter.  The 
students  can  use  the  system  on  an  opportune, 
time-available  basis  when  the  workload  permits. 


They  can  be  easily  monitored  to  ensure  training 
time  is  spent  productively.  Assistance  is  readily 
available,  should  they  have  questions  or 
problems.  Training  can  be  quickly  stopped,  if 
needed,  and  the  students  returned  to  work 
promptly.  If  the  students  must  travel  to  use  the 
system,  the  local  manager  will  sacrifice  both 
convenience  and  control. 

Determining  Necessary  Expansions  or 
Modifications.  If  the  systems  are  available  and 
the  location  is  conducive  to  training,  determine 
if  any  modifications  or  expansions  are 
necessary.  Some  modifications  are  likely. 

Are  additional  input  devices  required,  such  as 
touchscreen,  mouse,  track  ball  or  light  pen?  Be 
aware,  all  mice  are  not  compatible,  most  track 
balls  operate  like  an  upside  down  mouse,  and 
some  monitors  cannot  be  readily  modified  to 
add  a  touchscreen.  You  should  also  ensure  that 
the  available  or  selected  input  device(s)  are 
compatible  with  the  selected  authoring  system. 
Note  that  most  ICW  authoring  systems  will 
allow  either  multiple  input  devices  or  the 
selection  of  different  ones.  However,  this  may 
require  significant  time  and  effort,  and  may  not 
be  readily  done  at  the  training  site.  So,  this 
might  require  distribution  of  two  or  more 
different  versions  of  the  ICW.  Some  authoring 
systems  support  more  different  types  of  input 
devices  than  others  and  all  input  devices  are  not 
created  equal.  Mice  are  generally  the  most 
accurate  pointing  devices,  followed  by  the 
keyboard,  light  pen  and  touchscreen  in  that 
order. 

As  a  general  rule,  ICW  developed  for  a  less 
accurate  input  device  can  be  adapted  for  a 
higher  accuracy  input  device,  but  not  vice  versa 
-  without  a  significant  amount  of  effort.  Ease 
of  initial  learning  and  continued  use  should  also 
be  considered.  Mice  and  track  balls  may 
require  some  initial  student  orientation,  for  both 
hand/eye  coordination  and  how  to  make 
selections  by  "clicking  the  correct  button. " 

Both  light  pens  and  touch  screens  use  a  natural 
pointing  motion  and  are  easy  to  learn  to  use. 
But,  students  will  likely  complain  of  tired  arms 
when  required  to  touch  the  screen  for  extensive 
periods  of  time.  Keyboards  require  some 
typing  proficiency  or  the  students  will  likely  get 
frustrated  -  initially  learning  the  location  of  all 
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the  keys  and  the  functioning  of  special  control 
or  function  keys  can  be  overwhelming. 

Are  color  monitors  and  color  graphics  cards 
required  for  any  system?  Note  that 
monochrome  systems  are  appropriate  only  if 
there  are  virtually  no  complex  graphics  required 
in  the  ICW.  They  cannot  support  still  or 
motion  video.  Do  any  systems  require  printers 
to  print  necessary  training  reports?  Do  any 
need  additional  storage  devices?  Are  additional 
expansion  slots  or  communications  ports 
required  to  connect  the  input  devices,  printers, 
or  storage  devices?  Be  sure  to  identify  all 
required  connectors  and  cables  also. 

ICW  Presentation  Features  and 
Supporting  Media.  Before  you  can  make 
specific  hardware  determinations,  you  must 
thoroughly  evaluate  the  ICW  presentation 
features  required  to  support  your  training 
requirements.  These  presentation  features  are 
the  primary  determinants  of  the  specific 
hardware  configuration  and  the  authoring 
system  you'll  need.  Presentation  features  also 
drastically  impact  the  amount  of  computer  file 
storage  required,  a  primary  determinant  of 
distribution  medium. 

Audio.  If  audio  is  required  for  the  ICW,  note 
that  digital  audio  requires  very  large  amounts  of 
storage  space.  Digital  audio  is  not  really 
practical  for  distribution  via  floppy  diskettes  or 
for  use  on  average-sized  fixed  or  hard  disks. 
Note  also  that  most  vendors'  digital  audio  cards 
use  a  proprietary  file  storage  format  that  is 
incompatible  with  those  from  other  vendors. 

You  will  likely  need  to  purchase  the  same  type 
of  digital  audio  card  for  all  the  training  systems. 

Before  deciding  on  any  specific  audio  card, 
ensure  that  it  is  supported  by  several  different 
authoring  systems  (unless  you've  already 
decided  on  an  authoring  system).  Since  most 
digital  audio  cards  are  relatively  inexpensive 
and  provide  very  similar  capabilities,  you 
should  select  the  authoring  system  first,  then 
select  an  audio  card  that  it  supports.  Verify  that 
your  computer  monitors  have  built-in  speakers. 
If  not,  you  will  need  to  add  peripheral  speakers 
or  headphones.  You'll  also  likely  want  some 
type  of  optical  write  once,  read  many  (WORM) 
or  laser  ((either  videodisc  or  compact  disk 


(CD))  type  drive  to  support  adequate  quantities 
of  audio.  Similarly,  if  many  high-quality 
visuals  will  be  needed  in  the  ICW,  you'll  likely 
need  a  WORM  or  laser  type  drive.  You  may 
also  need  a  higher  resolution  graphics  card. 

Visuals  and  Audio  Using  Videodiscs.  Laser 
videodiscs  are  commonly  used  to  store  large 
quantities  of  visuals  and  audio  in  an  analog 
format,  with  54,000  video  frames  and  60 
minutes  of  monaural  audio  per  disc  side.  If  you 
intend  to  use  videodisc,  you  will  likely  need  to 
add  both  a  graphics  overlay  (i.e.  genlock)  card 
and  a  videodisc  player  to  each  system  since 
interactive  videodisc  (IVD)  systems  aren't  all 
that  common.  If  some  IVD  systems  are  already 
installed,  ensure  they  are  compatible,  or  plan  on 
replacing  at  least  some  components.  Note  that 
some  vendors  offer  both  color  graphics  and 
graphics  overlay  capabilities  on  the  same  card, 
while  others  use  a  two-card  set. 

Since  the  color  graphics  card  is  the  most 
common  source  of  incompatibility  in  IVD 
systems,  this  is  an  excellent  opportunity  to 
increase  compatibility  by  replacing  some  of  the 
existing  graphics  cards.  But,  make  sure  the 
type  you  are  considering  is  conq)letely 
compatible  with  both  the  existing  computer 
hardware  and  software.  Similar  to  specifying  a 
specific  printer  during  the  initial  setup  of  a 
word  processor,  some  authoring  systems  and 
languages  allow  you  to  specify  different 
graphics  cards  when  you  develop  the  ICW. 
However,  most  will  not  allow  you.lQ  change  the 
ICW  to  specify  a  different  graphics  card  once 
the  ICW  is  developed. 

Most  ICW  authoring  systems  or  languages 
allow  you  to  easily  specify  different  videodisc 
’’drivers*  to  support  the  common  models  of 
videodisc  players.  However,  this  may  or  may 
not  be  easily  accomplished  at  the  training  site, 
once  you  have  developed  the  ICW,  It  depends 
on  the  specific  authoring  language  or  system. 

Most  videodisc  players  use  a  standard 
Laservision  8-inch  or  12-inch  storage  format,  so 
incompatibility  is  usually  not  a  problem  with 
the  videodisc  itself.  There  are  other  less- 
common,  incompatible  videodisc  formats  used 
by  some  vendors  for  their  "one-time  recordable’ 
discs.  While  beneficial  for  ICW  prototyping 
and  development,  these  optical  media-directly 


recordable  (OMDR)  formats  are  not  frequently 
used  for  student  stations  due  to  the  high  per 
copy  cost  of  each  videodisc.  In  contrast, 
Laservision-format  videodiscs  are  mass 
replicated  at  a  much  lower  cost  per  copy,  when 
done  in  quantity.  The  OMDR  videodiscs 
cannot  be  mass  replicated. 

Even  with  Laservision  videodisc  players  there 
are  several  proprietary  (i.e.  incompatible) 
features  that  can  be  used  either  in  recording  the 
videodisc  (i.e.  digital  audio  tracks,  still  frame 
or  compressed  audio,  or  digital  data)  or  when 
authoring  the  ICW  (i.e.  'instant  jump*  features, 
video  frame  buffering,  etc.).  While  some  of 
these  features  are  enticing,  especially  digital 
audio  or  data,  weigh  their  value  against  the 
inherent  incompatibilities  very  closely. 

Avoiding  the  use  of  these  features  and  using 
only  the  standard  Laservision  features  will 
ensure  compatibility  of  the  ICW  with  all 
Laservision  videodisc  players.  As  a  final  note 
on  compatibility  of  videodisc  systems,  be  aware 
that  many  of  the  newer  computer  monitors  do 
not  have  speakers  built  in.  When  adding 
videodisc,  it  may  be  necessary  to  plug 
headphones  or  a  small  set  of  amplified  speakers 
into  the  headphone  jack  on  the  videodisc  player. 

Visuals  and  Audio  Using  Compact  Disks 
(CDs).  The  smaller,  4.72-inch  laser-read  CDs 
are  similar  in  form  to  videodisc,  but  use  a 
variety  of  digital  formats,  each  with  its  own 
acronym  and  intended  use,  including  CD-ROM, 
CD-A,  CD-ROM/XA,  CD-I,  DVI,  etc.  This 
storage  technology  is  changing  very  rapidly  and 
a  definitive  discussion  of  the  different  formats  is 
far  outside  the  scope  of  this  paper.  There  are 
several  general  considerations  of  CDs  that  are 
relevant,  however.  CDs  are  less  expensive  to 
replicate  than  videodiscs,  and  the  CD  drives  are 
also  less  expensive  than  videodisc  players.  CDs 
can  store  very  large  quantities  of  digital 
information,  either  data,  audio  or  visuals  for 
ICW  presentations.  All  require  a  CD  drive  and 
some  type  of  interface  card  in  the  computer, 
other  than  those  normally  used  for  hard  disks 
and  floppy  drives.  Make  sure  you  understand 
the  different  formats  thoroughly  before  deciding 
to  use  CDs. 

CD  Compatibility.  When  storing  only  data, 
the  CD  holds  approximately  550Mb,  and  is 


usually  compatible  with  all  CD-ROM  drives, 
which  are  normally  found  only  in  computers. 

As  of  the  date  of  this  paper,  any  CDs  that  are 
recordable  and  erasable  are  not  compatible  with 
standard  CD-ROM  drives.  While  some  CD- 
ROM  drives  will  also  play  the  CDs  used  for 
home  stereos  (CD-Audio,  OR  CD-A),  CD-A 
drives  will  not  play  CD-ROMs.  If  you  can  play 
a  CD-A  disk  on  your  CD-ROM  drive, 
headphones  or  amplified  speakers  plugged  into 
the  headphone  jack  on  the  drive  itself  are  often 
required,  since  there  may  be  no  provisions  to 
send  this  audio  into  the  computer  or  out  of  any 
speakers  in  the  monitor. 

Audio  can  also  be  stored  on  a  standard  CD- 
ROM  in  several  different  levels  of  quality  by 
using  an  audio  digitizing  card  in  the 
development  computer  and  a  digital  audio  card 
in  each  training  system.  While  this  allows  you 
to  store  a  mixture  of  conq>uter  text,  graphics 
and  audio  on  the  same  CD-ROM,  the  file 
format  of  the  audio  files  may  be  incompatible 
due  to  different  vendors  and  their  proprietary 
formats  for  the  different  digital  audio  files,  as 
mentioned  previously. 

Before  deciding  to  use  CDs  to  store  or 
distribute  your  KTW,  you  should  again  ensure 
that  the  authoring  system(s)  support  it.  As  of 
the  date  of  this  paper,  many  do  not.  CD- 
ROM/XA  is  a  relatively  new  atteiiq>t  to  address 
this  incompatibility  by  standardizing  the  audio 
cards  and  file  formats  for  this  media  mixture, 
using  the  standard  CD-ROM  formaLand  adding 
standard  extensions  for  audio.  CD-ROM/XA 
uses  an  additional  interface  card  that  may 
combine  these  two  functions,  control  of  the  CD 
drive  and  decoding  the  audio.  CD-ROM/XA 
disks  will  not  play  audio  on  systems  with  just 
standard  CD-ROM  drives,  such  as  those 
available  on  the  Desktop  III,  unless  an 
additional  audio  card  is  added.  ICW  authoring 
systems  or  languages  that  fully  incorporate  CD- 
ROM/XA  are  presently  very  limited. 

Constraints  of  CD-ROM.  All  CD-ROM 
disks  have  the  potential  to  store  very  high 
quality  still  visuals,  either  those  drawn  via 
computer  graphics  packages,  or  digitized  images 
from  a  scanner  or  video  camera.  Since  these  are 
truly  digital  images,  a  special  overlay  or 
genlock  card  is  not  required  in  the  computer  as 
it  is  for  videodiscs.  Note,  however,  that  the 
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graphics  card  in  the  computer  must  be  capable 
of  displaying  at  least  256  simultaneous  colors  in 
order  to  provide  acceptable-quality  digitized, 
color  images.  This  may  require  that  you  either 
add  additional  memory  (VRAM)  to  the  existing 
graphics  card  or  replace  it,  and  perhaps  the 
monitor  also  if  it  was  an  ’old”  graphics  card. 
Also,  note  that  the  higher  the  quality  of  the 
visual  in  terms  of  lines  of  resolution  and 
number  of  colors,  the  slower  the  picture  will 
’paint’  on  the  screen.  Also,  access  time  for  all 
CD  drives  is  much  slower  than  either  hard  disks 
or  floppy  diskettes. 

Before  deciding  to  use  CD-ROM  for  your  ICW, 
test  the  speed  of  CD-ROM  graphics  on  your 
’target’  system.  You'll  likely  find  that  while 
CD-ROM  can  be  a  very  efficient  distribution 
medium,  it  may  not  be  practical  to  operate  the 
ICW  directly  from  the  CD.  You  may  need  to 
copy  the  ICW  onto  a  hard  disk  first,  to  provide 
adequate  computer  response  time.  If  so,  make 
sure  the  hard  disks  are  large  enough  with 
enough  "free  space’  to  hold  the  ICW.  Digitized 
images  are  huge! 

Motion  Video  on  CDs.  When  it  comes  to 
motion  video,  the  CD  formats  are  even  more 
dynamic  and  confusing.  There  are  at  least  two 
CD  formats  that  store  motion  video,  as  of  the 
date  of  this  paper.  These  are  Digital  Video 
Interactive  or  DVI  and  Compact  Disk 
Interactive  or  CD-I.  DVI  is  a  proprietary 
product  of  Intel  Corporation  and  can  be  added 
as  a  set  of  boards  to  MS-DOS  microcomputers. 
DVI  can  play  full-screen,  full-motion  (i.e.  30 
video  frames  per  second)  video,  though  it  is 
currently  limited  to  240  lines  of  vertical 
resolution.  CD-I  is  a  published  standard  by 
Phillips  and  Sony,  and  should  soon  be  widely 
marketed  by  several  vendors  as  a  ’closed 
system’  that  contains  its  own  computer, 
graphics  card,  etc.  CD-I  carmot  be  added  to  an 
existing  microcomputer,  nor  can  it  currently 
provide  full-screen,  full-motion  video  (though  it 
should  be  available  before  the  product  is 
actually  marketed). 

There  are  "rumors"  of  further  CD-ROM/XA 
extensions  that  will  support  full-screen,  full- 
motion  video  -  while  motion  video  in  a  small 
part  of  the  screen  has  been  demonstrated.  Other 
pkoducts  have  been  announced  that  either  can 
play  partial-screen  motion  video  or  full  screen 


motion  at  less  that  30  frames  per  second. 
Authoring  systems  for  any  of  these  motion 
video  CD  formats  are  scarce,  if  available  at  all. 
Developers  also  report  at  least  twice  the 
development  time  required  as  for  videodisc.  If 
you  think  you  may  want  to  use  CDs  and  your 
ICW  requires  motion  video,  evaluate  the 
technical  qualities  of  the  system  (lines  of 
resolution,  number  of  simultaneous  colors, 
frame  speed  of  the  presentation,  size  of  the 
video  on  the  screen,  etc.)  very,  very  closely. 
Also,  potential  incompatibilities  abound  due  to 
the  different  formats.  While  these  technologies 
hold  very  signihcant  potential,  there  will  likely 
be  "format  wars’  similar  to  the  recent  one  in 
consumer  videotape  formats.  Are  you  sure  that 
you  can  "pick  the  winner?" 

Other  Considerations  Impacting  the 
Storage  and  Distribution  Medium. 
Though  the  ICW  presentation  features  required 
are  the  primary  determinants  of  the  storage  and 
distribution  medium,  other  factors  impact  this 
decision.  The  primary  ones  are  how  many 
copies  of  the  ICW  are  needed  and  how  often 
will  the  ICW  be  updated.  The  remaining 
factors  may  be  important  in  your  specific 
training  environment. 

Permanence  Versus  Updates.  Currently 
almost  all  the  very  large  capacity  storage  media 
are  essentially  disposable.  If  they  can  be 
reproduced  in  quantity  economically,  they 
cannot  be  erased  and  reused.  For  example, 
erasable  CDs  have  been  annoimcedund  there 
have  been  "rumors’  of  erasable  videodiscs  for 
years  -  none  are  currently  available  in  standard 
formats.  (There  are  currently  large  capacity 
magneto-optical  disks  that  can  be  erased  and 
reused,  but  they  are  relatively  new, 
incompatible  from  vendor  to  vendor,  and  there 
is  little  software  or  authoring  system  support  for 
them.) 

Some  formats  such  as  WORM  or  OMDR  disks 
can  added  to,  by  recording  additional 
information,  but  the  original  information  cannot 
be  changed  once  recorded.  They  offer  increased 
flexibility  for  prototyping,  initial  development, 
and  updates  -  bi't  the  media  is  more  expensive 
per  copy  and  they  cannot  be  mass  replicated. 

So,  you  must  carefully  consider  both  the  initial 
development  costs,  flexibility  required,  cost  per 
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copy  in  volume  (including  any  mastering  fees), 
and  the  amoimt  of  updates  anticipated. 

If  updates  will  not  be  frequent,  the 
'permanency*  of  the  recorded  medium  is  not 
very  important.  If  flexibility  is  important  and 
updates  are  frequent,  the  WORM  or  OMDR 
formats  are  cost  effective  in  small  quantities.  If 
large  quantities  are  required,  you  should  either 
use  a  CD  or  Laservision  videodisc  and  plan  for 
updates.  CDs  are  usually  ”re-replicated  and 
replaced,  ”  due  to  the  low  cost  per  copy.  ICW 
using  Laservision  videodiscs  can  usually  be 
updated  via  changing  the  software  stored  on 
floppy  diskettes  -  up  to  a  certain  point,  then 
replication  of  a  new  videodisc  is  required. 

(Note  that  if  the  computer  software  that  controls 
the  videodisc  is  also  stored  on  the  videodisc  as 
digital  data,  you  may  not  be  able  to  update  it 
using  floppies.  Replication  of  a  new  videodisc 
may  be  required  -  another  reason,  besides 
incompatibility,  to  avoid  this  practice.)  In  any 
case,  plan  for  the  updates  that  will  be  required 
and  consider  the  continued  mastering  and 
replication  costs  when  making  your  decision. 

Type  of  Use.  How  the  ICW  will  be  used  by  the 
student  also  impacts  the  media  consideration. 
Must  all  the  courseware  be  loaded  and  resident 
on  the  training  system  at  all  times?  This 
determines  the  volume  of  storage  needed  "on¬ 
line.  *  Or,  can  the  ICW  be  loaded  as  required? 

If  so,  how  long  does  it  take  to  load  it?  How 
fast  must  the  storage  device  be  accessed  to  fmd 
and  present  the  next  screen  to  the  student? 

Hard  disks  are  the  fastest,  followed  by  floppy 
diskettes.  Far  slower  are  videodiscs,  WORMs 
and  CDs,  not  necessarily  in  that  order.  Note 
that  the  "geography"  of  the  physical  location  of 
the  information  on  the  disk  and  the  way  it  is 
indexed  in  order  to  find  it  are  prime 
determinants  of  access  time.  For  example,  if 
the  next  video  still  that  is  needed  from  a 
videodisc  is  located  very  close  to  the  current 
one  (i.e.  plus  or  minus  100  frames  or  so)  the 
access  time  is  virtually  instantaneous.  If  it  is 
located  at  "the  other  end  of  the  disc”  it  can  take 
one  to  three  seconds  or  more. 

With  CDs  the  index  is  crucial,  and  the  time  for 
the  screen  to  "paint  or  draw”  is  also  a  factor.  In 
any  case,  you  will  likely  want  to  "mix"  the 
distribution  media,  by  using  floppy  diskettes  to 
distribute  the  control  software,  with  the  audio 


and  visuals  stored  on  a  CD  or  videodisc.  This 
media  mix  significantly  increases  the  ability  to 
make  quick,  easy  and  cost-effective  updates. 

The  floppies  can  be  loaded  to  the  conqruter's 
hard  disk,  if  desired,  and  the  CD  or  videodisc 
"swapped*  by  the  student  whra  needed. 

Systems  that  contain  multiple  CD  or  videodisc 
drives,  or  sinqrly  store  multiple  disks  and 
change  them  automatically  whm  needed  (often 
called  jukeboxes),  are  also  available.  But,  they 
may  be  cost-prohibitive  and/or  not  supported  by 
your  authoring  system. 

Distribution-Only  Media.  Some  storage  media 
are  appropriate  only  for  distributing  the  ICW 
and  cannot  be  readily  used  to  present  the  ICW 
to  the  student.  Magnetic  tape  or  tape 
cartridges,  regardless  of  the  specific  format,  fall 
into  this  category  since  they  either  are  not 
random-access  or  the  access  times  are 
prohibitively  slow.  They  also  wear  and  become 
unreliable.  Remote  communication  via 
networks  or  modems  has  virtually  hdl«i  into 
this  category,  given  the  advent  of  low  priced 
microcomputers  and  the  increasing  costs  of 
telephone  or  other  communications  lines. 
Remote  communications  can  be  very  effective  to 
distribute  ICW  and  to  report  student  data,  but 
are  being  used  less  and  less  to  actually  present 
the  ICW. 

ICW  Configuration  Control.  A  potentially 
crucial  consideration  is  the  configuration 
control  of  the  ICW  at  the  training  site.  You 
must  address  the  issues  of  what  happens  to  the 
ICW  once  you  identify  that  a  change  is 
required,  and  how  you  will  implement  that 
change.  For  example,  is  there  potential  for  a 
change  in  a  practice  or  procedure  that  may  cause 
barm  to  the  student  or  damage  to  equipment  if 
the  student  is  trained  incorrectly?  If  that  is  the 
case,  you  must  have  provisions  to  either  remove 
this  training  material  from  the  system  or  block 
access  to  it  as  soon  as  possible.  If  the  change  is 
important,  but  not  quite  as  critical,  you  can 
simply  notify  the  student  of  the  change  and 
perhaps  make  them  acknowledge  it  -  similar  to 
reading  or  posting  an  errata  sheet  for  written 
materials.  Other  changes  may  not  even  warrant 
notifying  the  student.  Whatever  the  case,  you 
must  have  provisions  to  make  these  changes 
while  retaining  as  much  pertinent  student  data 
and  student  status  in  the  ICW  as  possible.  We 
have  required  field  sites  to  return  videodiscs  to 


US  while  we  developed  a  change  in  the  ICW, 
just  to  make  sure  that  unsafe  training  did  not 
occur. 

Change  in  Student  Status.  A  change  to  the 
course  content  may  impact  the  training  status 
for  students  that  are  currently  "em-olled"  in  the 
ICW  and  students  who  have  previously 
completed  it.  For  example,  some  changes  may 
be  significant  enough  to  require  the  student  to 
repeat  the  particular  section  of  training  even 
though  they  had  previously  completed  it.  Or,  it 
may  require  that  only  the  students  that  are 
currently  enrolled  must  repeat  it  in  order  to  get 
"credit"  for  the  new  version  of  the  course.  Or, 
the  changes  may  not  impact  student  status  at  all. 
Regardless  of  the  criticality  of  the  change  in  the 
ICW,  you  also  need  to  address  provisions  to 
ensure  that  only  the  most  current  version  of  the 
course  is  in  use.  All  these  issues  should  be 
addressed  when  considering  the  distribution  or 
storage  media.  If  the  ICW  is  being  completely 
stored  on  a  hard  disk  at  a  remote  site,  can  you 
know  for  sure  that  previous  versions  were 
destroyed  and  only  the  current  version  is 
available? 

Replication.  Who  will  replicate  the  ICW, 
including  all  printed  adjunctive  materials?  CDs 
and  Laservision  videodiscs  must  be 
commercially  mastered  and  replicated,  if  large 
quantities  are  required.  Is  there  an  existing 
requirements  contract  for  replication  or  must  a 
contract  be  completed  each  time?  This  impacts 
both  price  and  lead  time.  OMDRs  and 
WORMs  can  be  locally  replicated,  but  each  one 
is  essentially  an  "original"  copy  recorded  in 
real-time,  so  it  is  very  time  consuming.  Hard 
disks,  even  removable  ones,  are  fast  but  are  not 
usually  practical  as  a  distribution  medium  due 
to  cost.  Each  copy  would  then  be  an  "original", 
recorded  one  at  a  time.  Magnetic  tapes  of  any 
format  are  inexpensive,  very  slow  to  copy,  and 
must  be  copied  one  at  a  time.  High  speed 
replicators  for  floppy  diskettes  are  readily 
available  and  inexpensive.  Or,  you  can  have 
them  replicated  commercially. 

If  the  ICW  will  be  distributed  over 
communications  lines,  there  will  be  no  direct 
replication  required.  But,  a  "bulletin  board 
service"  including  a  computer  system,  software, 
and  someone  to  "tend  and  maintain  it"  may  be 
required.  There  will  likely  be  communications 


costs,  if  only  a  one-time  cost  to  buy  the  network 
interface  cards  or  modems  required.  Use  of 
commercial  phone  lines  would  add  a  recurring 
cost.  Regardless  of  what  media  you  plan  to 
use,  be  sure  to  address  replication  requirements. 
Obviously,  the  greater  the  number  of  copies  of 
ICW  distributed,  the  greater  the  potential 
impact. 

Replacement  Due  to  Damage  and  Wear. 
Finally,  how  durable  is  the  media?  Consider 
also  the  fragility  of  the  media  and  its 
susceptibility  to  damage.  You  should  also 
consider  any  required  replacement  of  the  media 
due  simply  to  wear.  For  example,  CDs  and 
videodiscs  that  have  been  mass-replicated  are 
very  durable,  though  they  can  warp  or  break, 
and  are  not  susceptible  to  wear.  Those 
produced  by  direct  read  after  write  (DRAW) 
technologies  are  more  fragile,  can  wear,  and 
deteriorate  over  time  due  to  exposure  of  the 
recording  media  to  light  and  air.  With 
continued  use,  replacement  at  some  point  is 
inevitable.  Or,  consider  floppy  diskettes.  The 
3.S  inch  diskette  protects  the  recording  media 
with  a  shutter  that  automatically  closes  upon 
removal  from  the  computer.  The  5.25  inch 
floppies  leave  the  recording  media  exposed  and 
susceptible  to  accidental  damage  and  increased 
wear  due  to  dirt,  dust,  etc.  Regardless  of  how 
the  ICW  will  be  replicated,  be  sure  to  include 
all  replication  costs  (money  and  time)  for  initial 
distribution,  replacement  and  updates  in  your 
media  considerations. 

Packaging,  Storage  and  Distribution.  The 
final  consideration  deals  with  the  packaging, 
storage,  and  distribution  of  the  ICW.  Is  the 
packaging  readily  available?  Does  it  adequately 
protect  the  ICW  during  shipment  and  storage? 

Is  a  "bench  stock"  of  stored  ICW  required  for 
quick  shipment  or  can  it  be  replicated  as 
needed?  For  example,  mailers  are  readily 
available  for  shipping  floppy  diskettes  and 
storage  isn't  a  real  problem.  They  can  be 
replicated  as  needed  from  master  copies  that  are 
safely  stored. 

On  the  other  hand,  both  CDs  and  videodiscs 
require  payment  of  a  fairly  expensive  mastering 
and  setup  fee  each  time  they  are  replicated  -  or 
you  must  pay  the  replication  contractor  to  store 
the  master  copy  in  their  protected  enviromnent, 
then  pay  only  the  cost  per  copy.  Most  people 
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try  to  anticipate  their  future  needs,  then 
replicate  and  store  enough  "bench  stock" 
initially  to  meet  their  anticipated  requirements. 
In  comparison,  WORM,  DRAW  or  OMDR 
technologies  allow  you  to  replicate  when 
needed,  but  you  incur  both  the  setup  and 
replication  time,  in  addition  to  having  to  store 
these  fragile  media  in  their  protective  covers  in 
a  suitable  environment.  Consider  any  costs 
associated  with  packaging  and  storage  for  the 
potential  media,  including  both  storage  space 
and  environmental  requirements.  The 
differences  in  actual  cost  of  shipping  the 
different  media  will  not  likely  be  significant 
enough  to  warrant  consideration  unless  the 
volume  of  shipments  is  very  high.  Note, 
however,  if  the  ICW  is  distributed  via 
communications  systems  there  are  no  shipment 
costs  other  than  any  recurring  communications 
charges. 

Final  Note  on  Hardware.  In  summary, 
both  hardware  availability  and  compatibility  are 
interrelated  and  critical  since  you  will  have  to 
develop  the  ICW  to  operate  on  the  "lowest 
common  denominator"  of  hardware  systems. 
That  is  why  you  must  carefully  and  concisely 
define  the  common  denominator  -  and  what 
equipment  expansion  or  modification  is  required 
for  existing  systems  to  bring  them  up  to  a  new 
common  denominator  that  will  meet  your 
training  requirements. 

Make  no  assumptions  and  take  nothing  for 
granted,  especially  when  dealing  with 
compatibility.  For  example,  we  selected  and 
standardized  on  a  particular  brand  and  model  of 
IVD  system  to  ensure  compatibility,  since  it 
offered  a  relatively  simple,  integrated  system. 
But,  we've  been  forced  to  make  updates  in  some 
ICW  because  the  manufacturer  changed  the 
version  of  the  basic  input-output  system  (BIOS) 
chip  in  their  computers  in  the  middle  of  a 
production  run,  even  though  everything  else 
remained  the  same.  Only  after  the  ICW  was 
implemented  in  the  field  did  we  discover  the 
ICW  operated  just  fine  with  the  older  BIOS 
chip,  but  there  were  intermittent  errors  when 
operating  with  the  new  BIOS  chip. 
Consequently,  we  now  thoroughly  test  ail  the 
ICW  on  three  different  configurations  of  the 
vendor's  systems  prior  to  shipment. 


Courseware  Design  Requirements. 

The  design  of  exportable  ICW  is  more  complex 
and  challenging  than  that  of  ICW  to  be  used  in  a 
classroom  environment.  You  must  plan  for 
minimal  assistance  to  be  available  to  the  student 
at  the  training  site,  and  design  the  ICW 
accordingly.  Consequently,  exportable  ICW 
will  likely  require  greater  design  and 
development  time. 

Comprehensiveness.  First,  the  ICW  must  be 
"self-contained"  as  much  as  possible. 

Documents  such  as  technical  orders  or  job 
guides,  that  you  can  safely  assume  will  be 
available  at  the  training  site,  can  simply  be 
referenced  in  the  ICW.  However,  they  should 
be  treated  as  reference  material  only  -  you 
probably  shouldn't  require  that  they  be  used  in 
order  to  complete  the  training.  Otherwise,  if 
those  reference  materials  are  being  used 
elsewhere  at  the  training  site,  no  training  can  be 
accomplished.  If  any  other  type  of  written  or 
other  adjunctive  materials  are  required  for  the 
student  to  complete  the  ICW,  then  they  should 
be  developed  and  distributed  as  a  package  with 
the  ICW. 

Evaluate  very  closely  any  requirement  for 
"expendable"  documents  such  as  workbooks 
that  must  be  completed  by  the  student.  They 
require  that  someone  at  the  training  site 
reproduce  (unless  you  provide  them),  stock,  and 
issue  them  to  each  student.  Further,  they  are 
easily  lost  or  misplaced  by  the  student,  which 
can  cause  delays  or  "missed  opportunities'  for 
training.  The  student  will  naturally  be  reluctant 
to  get  another  workbook  and  start  all  over, 
should  they  forget  or  misplace  theirs.  As  a 
general  rule,  any  increase  in  administrative 
workload  placed  on  management  at  the  training 
site  will  likely  decrease  the  utilization  of  the 
ICW. 

Further,  the  ICW  must  contain  more  helps, 
hints,  glossaries,  etc.,  than  ICW  used  in  a 
classroom.  There  will  not  be  an  instructor  or 
anyone  else  to  supervise  the  trainee  at  all  times, 
providing  assistance  and  answering  questions. 
Similarly,  more  extensive  explanations  will  be 
required  in  the  ICW,  especially  for  remediation 
or  feedback  to  incorrect  student  responses.  You 
should  try  to  anticipate  potential  student 
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questions  and  mistakes  and  design  provisions 
for  them  in  the  ICW. 

The  ICW  must  also  be  both  more  diagnostic  and 
more  prescriptive.  There  will  be  no  instructor 
to  evaluate  the  students'  progress  and  prescribe 
further  training,  review  or  remediation.  You 
cannot  count  on  the  supe  'visor  or  trainer  to  be 
actively  involved  in  managing  the  student's 
training  either.  Diagnostic  pretests  and  frequent 
review  exercises  or  progress  checks  should  be  a 
standard  part  of  your  instructional  design. 

Your  ICW  design  should  be  preemptive,  trying 
to  diagnose  and  prevent  the  student  from 
making  mistakes  as  much  as  possible,  rather 
than  dependent  on  later  evaluation  and 
remediation. 

Complexity.  Meanwhile,  you  must  also  plan 
for  the  students  to  have  a  greater  range  of  entry- 
level  expertise  and  knowledge  and  include 
accommodations  in  the  ICW.  The  relevant 
work  experience  of  the  students  will  likely  vary 
much  greater  than  those  sent  to  classroom 
training,  especially  initial  skills  training.  For 
example,  even  though  you  may  direct  course 
prerequisites  and  diagnose  their  mastery  in  a 
prerequisite  test,  how  will  you  treat  the  student 
who  has  already  completed  the  course  and  wants 
to  merely  review  a  section  or  take  it  as  a 
refresher?  You  definitely  do  not  want  to 
discourage  this  type  of  utilization,  since  it  is 
one  of  the  unique  strengths  of  exportable  ICW. 
Instead,  you  should  design  provisions  for 
multiple  paths  through  the  ICW  based  on  pretest 
scores  or  their  progress  through  previous 
material. 

As  a  general  rule,  it  is  more  desirable  to  present 
the  student  with  "too  tough  a  challenge" 
initially  than  to  bore  them  with  material  they 
already  know  or  that's  too  simplistic.  If  you 
overestimate  the  student's  entry-level 
knowledge,  you  can  still  identify  their 
weaknesses  during  subsequent  progress  checks 
or  review  exercises  and  provide  remediation. 

On  the  other  hand,  if  the  trainee  is  bored  by  the 
ICW  being  "too  ea.sy",  you  may  never  know  it 
unless  it  appears  in  course  critiques  or  indirectly 
in  low  ICW  utilization.  Challenges  can  be 
motivating  -  boredom  is  almost  always 
demotivating! 

It  should  al.so  be  very  ea.sy  for  the  student  to 


skip  material  that  they  have  already  mastered  or 
that  is  redundant.  However,  research  indicates 
that  the  students  may  not  be  qualified  to  best 
assess  their  own  capabilities  or  to  determine  a 
sequence  of  training.  You  should  give  the 
students  as  much  freedom  of  movement  within 
the  ICW  as  possible,  yet  their  instructional 
sequence  should  be  determined  by  proven 
mastery  of  prerequisite  material.  Students 
should  then  have  virtually  total  freedom  to 
review  the  sections  of  the  course  as  desired, 
once  they  have  satisfactorily  completed  it. 

Impact  on  Development  Time.  All  these 
factors  relate  to  ICW  design,  development  and 
validation  time.  It  is  time  consuming  to  analyze 
the  subject  matter  to  the  point  of  determining 
conunon  mistakes  for  different  levels  of  skills 
and  knowledge,  then  design  provisions  for 
them.  It's  time  consuming  to  develop  multiple 
paths  through  the  ICW,  when  most  students  will 
not  see  them  all  and  many  will  progress  along 
the  shortest  path.  Yet,  all  paths  must  be 
thoroughly  tested  and  "de-bugged.*  Further,  it 
is  time  consuming  to  validate  the  accuracy  of 
your  diagnostic  tests  and  the  appropriateness  of 
the  instructional  prescriptions.  Similarly,  the 
relevance  and  effectiveness  of  the  remediation 
may  be  difficult  to  validate  since  you  must 
compare  the  type  of  mistake  to  both  the  actual 
and  anticipated  cause  for  the  mistake.  The  mere 
existence  of  multiple  paths  means  that  more 
students  will  be  needed  to  "try  out"  the  ICW  in 
order  to  validate  all  possible  paths.  As  a 
general  rule,  an  increase  in  ICW  complexity 
results  in  exponential  increases  in  design, 
development  and  development  time. 

User-friendliness.  This  overused  term  relates 
to  the  student's  general  impression  of  the 
training  session,  how  they  relate  to  both  the 
ICW  and  the  hardware,  and  the  entire  strategy 
or  philosophy  used  to  create  the  ICW. 

Exportable  ICW  must  be  more  user-friendly, 
since  assistance  and  support  is  not  readily 
available  on  site.  Some  aspects  have  been 
previously  described,  including  both  hardware 
and  courseware  that  are  easy  to  learn  to  use, 
adaptability  of  the  ICW  to  the  individual 
student,  and  a  design  intended  to  prevent  and 
preempt  student  mistakes.  Other  aspects 
include  accuracy,  consistency,  reliability  and 
predictability  of  the  ICW's  appearance  and 


27 


operation.  It  applies  to  more  than  the 
courseware  itself,  but  also  the  hardware  system 
and  any  supporting  documentation. 

The  students  should  always  know  what  to 
expect  and  should  feel  that  they  are  in  control  of 
the  training  system  and  the  ICW  -  within  the 
limits  of  the  instructional  prescriptions  based  on 
their  performance.  The  goal  is  to  have  the 
students  so  comfortable  with  the  system  that 
they  are  no  longer  conscious  of  the  hardware  or 
how  they  are  interacting  with  it.  It  becomes 
almost  second  nature.  Their  entire  focus  can 
then  be  on  experiencing  and  assimilating  the 
course  content.  That  content  should  ideally  be 
presented  so  realistically  that  the  student  can 
'suspend  their  disbelieP,  and  'get  into  it”. 

The  true  power  of  multimedia  is  it's  ability  to 
address  multiple  senses  simultaneously,  thus 
increasing  the  realism  almost  exponentially. 

Design  Strategy.  How  is  this  emphasis  on  user 
friendliness  implemented?  At  the  macro  level, 
the  course  must  be  at  least  somewhat  adaptive  to 
the  individual  student,  providing  a  relevant  and 
realistic  training  experience.  At  the  micro 
level,  each  interaction  should  be  carefully 
designed  to  "draw  the  student  into"  the  training 
and  encourage  them  to  interact  with  the  content, 
not  the  training  system.  The  ICW  should 
respond  accordingly,  in  a  friendly, 
conversational  way.  The  response  should  be 
appropriate,  accepting,  and  encouraging. 

Imagine  a  conversation  with  another  person 
whose  response  to  each  of  your  statements 
makes  little  sense  in  this  context,  who 
intimidates  you  with  their  "know  it  all"  tone, 
and  who  will  volunteer  little,  responding  only 
to  your  direct  input,  while  pointing  out  each 
and  every  little  mistake.  Not  really  a  conducive 
atmosphere  for  learning,  is  it?  Yet  that  is 
exactly  how  much  ICW  is  designed.  Or,  the 
ICW  is  simply  an  automated  and  perhaps 
illustrated  book  that  the  student  mechanically 
pages  tl^rough,  answering  occasional  questions. 
"Page  turners'  are  fully  deserving  of  their 
notoriety! 

The  most  effective  ICW  designs  require  an 
immense  attention  to  detail  at  every  step.  Each 
screen,  each  interaction  is  carefully  designed 
and  created,  then  painstakingly  assembled  and 
tested.  Once  assembled,  it  is  evaluated 
holistically,  ideally  by  "outsiders".  What  theme 


does  it  contain?  What  tone  or  mood  does  it 
present?  How  does  it  look  and  feel?  Does  it 
seem  real?  What  impressions  are  you  left  with? 
What  strikes  you  about  it,  both  positively  and 
negatively?  Etc. 

Documentation.  This  same  attention  to 
perceptions  and  detail  apply  to  the  development 
and  testing  of  the  documentation  also.  Do  not 
rely  entirely  on  "on-line  help,  documentation  or 
tutorials. '  What  happens  when  the  ICW  fails  to 
"boot"  or  start  up,  so  these  features  are  not 
accessible?  There  should  always  be  a  'quick 
reference"  guide  on  paper  included  with  the 
ICW,  even  if  most  of  its  content  is  duplicated  in 
the  ICW.  In  case  of  questions,  it  is  often  far 
easier  to  "look  it  up  in  the  book"  instead  of 
loading  and  starting  the  ICW  to  review  it.  At 
the  same  time,  the  documentation  should  be 
short,  clear  and  concise.  Unnecessary  detail 
discourages  use  and  "hides'  any  truly  needed 
information. 

Develop  the  documentation  carefully  and 
validate  it,  just  as  you  do  the  actual  courseware. 
The  documentation  should  address  two  separate 
audiences,  the  student  and  the  manager  (be  it 
the  student’s  supervisor  or  someone  in  charge  of 
training).  Both  need  to  clearly  understand  the 
content,  basic  structure,  objectives  and 
performance  standards  of  the  ICW.  Both  need 
to  know  how  to  start  the  ICW,  be  aware  of 
common  problems  encountered  with  either  the 
ICW  or  training  system,  and  know  how  to  deal 
with  them.  Both  need  to  know  what  individual 
student  reports  are  available  from  the  system 
and  their  basic  content. 

The  manager  also  needs  to  know  how  to  access 
any  student  summary  reports  provided,  and 
their  content.  If  the  ICW  is  to  be  stored  on  a 
hard  disk,  the  manager  needs  to  know  how  to 
load  and  unload  it.  The  manager  also  needs  to 
know  how  to  perform  basic  "file  maintenance” 
on  the  ICW,  making  backup  copies  as  required, 
plus  how  to  add,  remove,  and  perhaps  edit  the 
students'  records.  If  utilization,  student 
performance,  or  test  item  analysis  data  is 
extracted  and  returned  to  the  development 
agency  for  analysis,  the  manager  needs  to  know 
how  to  extract  it. 
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Computer  Managed  Instruction 
(CMI)  Requirements.  This  discussion  of 
what  data  is  recorded  and  how  it  is  used  was 
delayed  to  this  point  to  reinforce  the  importance 
of  "user-friendliness"  here  also.  The  CMI 
should  also  generally  be  self-contained,  with 
minimal  intervention  required  by  local 
management  personnel.  Unless  there  is  a 
specific  reason  not  to  do  so,  allow  the  students 
to  register  themselves  to  enter  the  ICW  and  let 
their  performance  on  diagnostic  tests  determine 
their  instructional  sequence.  Don't  require  the 
manager  to  add  the  students  to  the  system,  or 
assign  the  lessons  that  they'll  take.  The  ICW 
should  make  it  easier  for  the  manager  to 
perform  his  job,  not  add  additional  tasks  to  his 
already  busy  schedule.  If  the  student  must  rely 
on  the  manager  to  intervene,  then  training  will 
be  dependent  upon  when  the  manager  can  "get 
around  to  it"  and  unnecessarily  delayed. 

Types  of  CMI  Data  Needed.  The  next 
consideration  is  the  type  of  student  data  that  you 
want  or  need  to  accumulate.  What  type  of 
student  performance  data  is  to  be  captured, 
merely  test  scores  and  registration  data,  or  a 
"screen  by  screen"  record  of  their  progress? 
How  often  will  the  data  be  returned  to  the 
training  developer(s)  for  analysis  and 
evaluation?  Must  the  student  data  be  reported 
to  another  agency  for  some  management 
purpose,  such  as  an  update  to  the  personnel  data 
systems  or  the  Community  College  of  the  Air 
Force?  Is  there  a  requirement  for  safeguarding 
either  privacy  act  data  or  the  integrity  of  your 
ICW  tests?  Since  these  CMI  functions  can  also 
impact  the  selection  of  the  training  system 
t^unflguration,  the  distribution  media,  and  the 
(iUthorinj  system,  the  requirements  should  be 
concisely  defined.  Specific  reporting 
requirements  to  outside  agencies  such  as  the 
personnel  system  often  dictate  some  aspects  of 
what  and  when  to  report. 

Further,  look  at  how  many  students  are 
anticipated  at  each  training  site  in  a  given  time 
frame.  Don't  overlook  the  impact  of  personnel 
turnover.  How  long  will  the  students'  records 
be  maintained  on  the  system  to  allow  them 
opportunities  to  review  previously  completed 
material?  What  test  item  and  ICW  validation  or 
evaluation  data  is  required? 


While  storing  only  test  scores  for  each  student 
will  likely  meet  all  requirements  for  student 
performance  and  completion,  additional 
information  is  often  required  for  the  ICW 
developer.  They'll  be  interested  in  each 
student's  response  to  each  test  item  for  use  in 
test  item  validity  and  reliability  evaluations. 
They  may  want  to  know  which  path(s)  through 
the  ICW  were  taken  for  each  student,  along 
with  their  test  scores,  to  evaluate  the  criteria 
used  for  their  instructional  prescriptions  and 
their  validity.  They'll  likely  want  the  students 
to  complete  a  subjective  critique  of  the  ICW, 
and  to  correlate  it  against  their  scores.  The 
amount  of  time  spent  in  training  is  valuable 
information  to  both  the  training  developer  and 
management  at  all  levels.  How  often  will  any 
of  this  data  be  extracted?  All  these  issues 
impact  how  the  CMI  data  will  be  stored  and  the 
volume  of  storage  necessary. 

Limiting  CMI  Data.  As  a  general  rule,  CMI 
data  requirements  should  be  limited  to  the 
minimum  of  "needed,  meaningful  data",  in 
order  to  minimize  the  storage  required  and  to 
provide  alternative  storage  media  and  locations. 
If  it  isn't  both  needed  and  significant,  then 
don't  store  it.  Note  however,  that  if  you  must 
err,  then  err  on  the  side  of  storing  "too  much" 
data.  Should  you  later  find  that  you  need  some 
data  that  hasn't  been  stored,  it  cannot  usually  be 
"recreated"  or  obtained  by  other  means.  Plan 
carefully,  yet  concisely! 

CMI  Data  Storage  Locations.  Note  that 
centralized  storage  of  CMI  data  on  a  single 
floppy  or  hard  disk  (as  opposed  to  a  floppy  per 
student,  for  example)  makes  it  much  easier  to 
manage  the  ICW  and  extract  any  needed  CMI 
data.  However,  this  is  not  an  ideal  solution  if 
there  are  multiple  training  systems  or  multiple 
copies  of  the  ICW  at  the  site.  If  there  are 
multiple  training  systems  at  the  site  with  the 
CMI  data  and  courseware  loaded  to  their  hard 
disks,  then  the  student  must  use  the  same 
system  to  complete  the  entire  course.  If  there 
are  multiple  systems  at  the  site  using  multiple 
copies  of  ICW  stored  on  floppy  diskettes,  the 
student  must  always  use  the  same  copy  of 
ICW. 

The  student  must  always  "report"  to  the  same 
CMI  data  files,  so  in  either  of  these  cases  it  may 
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be  easier  to  provide  a  floppy  for  each  student  to 
use  for  storing  their  records.  If  so,  consider  the 
cost  of  providing  all  these  floppies  and  the 
potential  for  the  student  to  lose  theirs.  You'll 
probably  want  to  require  that  the  students  store 
their  individual  floppies  in  a  centralized  location 
to  prevent  losing  them.  It  also  makes  them 
available  at  all  times  for  preparing  any 
management  reports. 

CMI  Data  Maintenance.  Regardless  of  how  or 
where  the  data  is  stored,  there  must  be 
provisions  to  ’deregister"  students  when  they 
depart  the  location  or  their  records  are  no 
longer  needed  for  whatever  reason.  Is  there  a 
requirement  to  extract  the  student's  records  and 
forward  them  to  another  organization  upon 
reassignment?  Since  the  CMI  data  will  be 
constantly  growing,  there  must  be  some  easy 
way  to  "clean  out’  other  old  CMI  data. 
Similarly,  there  must  be  an  easy  process  to 
extract  any  needed  CMI  data  for  provision  to 
the  training  developers  or  others. 

It  may  be  beneficial  to  differentiate  the  data 
stored  based  on  who  requires  the  information, 
then  store  it  in  different  files  or  locations  based 
on  intended  use.  This  makes  it  easier  to 
maintain  only  the  essential  data  "on-line"  while 
providing  for  easy  extraction  as  needed.  For 
example,  we  store  each  student's  response  to 
each  test  item  or  critique  question  in  a 
"centralized"  file  for  each  test  and  critique, 
along  with  their  student  number  for  cross- 
referencing.  We  store  their  overall  test  scores 
for  each  test  and/or  objective  in  a  single  file  for 
each  student,  along  with  other  background 
information  such  as  elapsed  time  in  training. 
Every  six  months  we  extract  the  CMI  data  from 
each  course  at  each  site  (using  a  "staggered" 
schedule)  for  evaluation  using  a  single 
automated  process. 

This  process  copies  to  an  empty  diskette  each 
student's  data  file,  each  test  item  analysis  file, 
and  the  critique  item  analysis  file.  Then,  the 
program  deletes  the  test  item  and  critique  item 
analysis  files  from  the  ICW’s  CMI  data 
diskette.  Each  student's  overall  status  in  the 
course  is  retained  by  their  unique  data  file, 
which  remains.  However,  the  detailed  test  item 
and  critique  item  analysis  data  is  removed  to 
prevent  "overcrowding"  the  space  allocated  to 
CMI  data.  The  new  diskette  containing  these 


files  is  returned  to  the  training  developers  for 
consolidation  with  the  CMI  data  from  other 
sites  and  subsequent  evaluation.  Because  the 
item  analysis  files  are  cross-referenced  by 
student  number,  we  can  make  any  necessary 
correlations  in  evaluating  either  the  ICW  or  its 
tests. 

Conclusion.  The  ultimate  success  of 
exportable  ICW  will  likely  be  determined  by 
how  well  it  was  planned,  considering  its  unique 
challenges,  and  how  well  that  plan  was 
executed.  Failures  cannot  likely  be  attributed  to 
any  inherent  shortcomings  in  the  media  or 
concept.  Exportable  ICW  has  no  one  on  site  to 
make  up  for  its  weaknesses,  expound  on  its 
merits,  or  make  excuses  for  its  failures.  It  must 
"live  or  die"  on  its  own  -  and  it  reflects  directly 
on  the  people  who  planned  it,  developed  it  and 
sent  it  out!  Plan  accordingly! 


KEYWORDS  1.  Exportable  training  2. 
Distance  education  3.  Interactive  courseware 
(ICW)  4.  Interactive  videodisc  (IVD)  5. 
Computer-based  training  (CBT)  . 


30 


A  ROLE  FOR  THE  PERFORMANCE  TECHNOLOGIST 


Frederick  L.  Norman 
U  S  AIR  FORCE  (ATC) 


Total  Quality  Management  (TQM)  is  a  moving  force  throughout 
industry  and  government  today.  Many  people  think  that  TQM  is  a 
vital  key  in  our  struggle  to  maintain  leadership  in  the  industrial 
world.  (Peters,  1987)  If  TQM  is  vital  then  it  must  be  implemented 
properly.  Implementation  of  TQM  is  most  successful  when  it  is  fully 
understood  and  properly  supported.  It  is  within  the  implementation 
part  of  TQM  that  the  performance  technologist  can  play  an  important 
role  in  assisting  in  the  success  of  any  TQM  effort.  This  paper  will 
suggest  how  the  performance  technologist  can  be  instrumental  in 
developing  the  training  in  the  use  of  the  TQM  tools  and  also  assist  in 
identifying  performance  problems  after  TQM  has  been  implemented. 
Before  starting  that  discussion  some  definitions  of  TQM  and  some 
other  background  information  vill  be  provided. 

Definitions.  Quality  has  been  defined  as:  Meeting  the  customers 
requirement  the  first  time  and  everytime. (Western  Executive 
Seminar  Center,  1990)  Also  Quality  has  been  defined  as:  Doing  the 
right  thing,  right  the  first  time,  on  time,  all  the  time;  always 
striving  for  improvement  and  always  satisfying  the  customer(DoD 
TQM  A  Guide  for  Implementation,  1989).  Quality  is  also  defined  by 
the  Federal  Government  as:  The  extent  to  which  a  product  or  service 
meets  customer  requirements  and  is  fit  for  use.  (OBM  Circular  No.  A- 
132,  1988) 

Total  Quality  Management:  A  strategic,  integrated  management 
system  for  achieving  customer  satisfaction  which  involves  all 
managers  and  employees  and  uses  qualitative  methods  to 
continuously  improve  an  organization’s  processes.  (Federal  Quality 
Institute,  1 990) 

Another  definition  of  TQM:  A  process  that  taps  the  creativity  of  all 
employees  who  are  responsible  for  quality  products  and  services, 
from  top  management  to  line  employee,  to  achieve  the  ultimate  goal, 
customer  satisfaction. 


TQM  is  both  a  philosophy  and  a  group  of  guidelines  and  principles 
that  are  the  basis  for  a  continuously  improving  organization.  It  is  the 
application  of  quantative  methods  and  human  resources  that  result 
in  the  improvement  of  products  and  services  provided  for  a 
customer  and  the  extent  to  which  customers  needs  are  met. 


What  the  Leading  Quality  Experts  Say  About  TOM  Training.  Dr.  W. 
Edward  Doming  is  perhaps  most  widely  known  for  his  work  with  the 
Japanese  after  World  War  II.  He  is  a  statistician  and  he  began 
teaching  statistical  quality  control  in  Japan  in  the  1940’s.  He  is 
acknowledged  as  an  important  contributor  to  the  Japanese 
ascendency  in  quality  management.  Dr.  Doming  states  that 
companies  should  also  establish  constancy  of  purpose  by  means  of 
innovation,  research  and  education,  continuous  improvement  of 
products  and  services  and  maintenance  of  equipment  etc.  (Walton, 
1986).  He  is  also  noted  for  his  fourteen  points  for  management. 
Points  six  and  thirteen  are  "institute  training"  and  "institute  a 
vigorous  program  of  education  and  retraining."  (Walton,  1986). 

Dr.  Joseph  M.  Juran  has  most  likely  contributed  more  to  the  fields  of 
quality  control  and  management  as  all  other  contributors  combined. 
He  taught  the  Japanese  the  principles  of  quality  management  in  the 
1950's.  One  of  the  main  points  in  his  philosophy  is  the  need  for 
quality  training  to  stay  competitive.  According  to  Juran  "....many 
Japanese  companies  have  trained  100  %  o^  their  employees  in  the 
quality  disciplines.  Few  U.S.  companies  provide  quality  training  for 
more  than  five  percent  of  their  employees."  (Quality  Leadership, 
1989). 

Phillip  B.  Crosby,  author  of  Quality  is  Free  and  Quality  Without  Tears 
developed  the  Zero  Defects  program  and  founded  the  Crosby  Quality 
College.  The  essence  of  Crosby’s  quality  improvement  process  is 
embodied  in  what  he  calls  the  Absolutes  of  Quality  Management  and 
the  Basic  Elements  of  Improvement.  The  Absolutes  address  the 
question  of  what  quality  is  and  what  standards  and  systems  are 
needed  for  the  achievement  of  quality.  One  of  his  Basic  Elements  of 
Improvement  is  Quality  training  materials  and  instruction  must  be 
excellent. 

The  Tools  Used  for  Continuous  Improvemer^  An  important  part  of 
TQM  is  identifying  and  correcting  probL  ns.  There  are  some 
management  tools  that  are  used  in  the  TQM  process  to  identify  and 


correct  problems  and  make  continuous  improvements.  Tiie  tools 
used  for  continuous  improvement  can  be  divided  into  two  categories 
as  follows  (Brassard,  1985); 

Problem  identification; 

Flow  Charts 

Check  Sheets 

Brainstorming 

Nominal  Group  Techniques 

Problem  Analysis 
Histograms 
Scatter  Diagrams 
Control  Charts 
Process  Capability 
Force  Field  Analysis 

Some  tools  can  be  included  in  either  category.  Those  are: 

Pareto  Charts 
Cause  &  Effect  Charts 
Run  Charts 
Stratification 

There  are  several  process/performance  improvement  models  used 
for  TQM.  They  all  use  the  tools  identified  above  in  their 
process/perfermance  improvement  procedure.  They  also  all  follow 
the  Pian-Do-Check-Act  concept.  The  American  Productivity  & 
Quality  Center's  IMPACT  model  consist  of  six  steps  as  follows; 

Planning 

Assessment 

Direction  Setting 

Measures  Development 

Service  (RE)  Design  &  Implementation 

Results,  Review  &  Recycle 

Other  models  consist  of  seven  steps  that  are  similar  in  many  ways  to 
the  IMPACT  process.  What  thev  have  in  common  is  in  the  planning 
step.  This  step  almost  always  includes  the  requirement  for  training 
The  training  required  is  to  make  all  employees  aware  of  the  needs 
for  and  the  benefits  of  TQM,  and  train  them  in  the  use  of  the  tools 
and  techniques  to  support  continuous  improvement.  The  scope  and 


intensity  of  training  will  depend  on  such  factors  as  organizational 
level,  nature  of  work,  and  specific  processes  under  review  for 
improvement.  The  training  should  be  tailored  to  support  the  visions 
and  goals  set  by  top  management 

Performance  Technologist's  Role.  There  is  a  great  tendency  on  the 
part  of  most  people  when  hearing  of  TQM  to  think  of  it  as  just 
another  program.  Or  they  think  of  it  as  managements  latest  idea  on 
how  to  get  more  output  from  an  already  over  worked  workforce. 
These  false  ideas  are  indeed  difficult  to  overcome.  There  are  other 
more  difficult  tasks  associated  with  implementing  TQM  than 
correcting  false  ideas.  One  of  the  most  difficult  parts  in  adopting 
TQM  is  to  get  individuals  to  understand  that  TQM  is  a  change  in 
beliefs  and  subsequently  a  change  in  behavior.  Getting  individuals  to 
understand  that  TQM  is  a  change  in  beliefs  is  indeed  a  problem. 
Another  aspect  of  that  problem  is  to  get  individuals  to  change  their 
beliefs.  Causing  these  changes  in  beliefs  and  behavior  (or 

performance)  in  individuals  is  how  the  instructional  technologist  and 
the  performance  technologist  disciplines  can  be  effectively  used  in  all 
phases  of  a  TQM  effort.  All  people  can  change  regardless  of  age  and 
circumstance  (Tough,  1982).  When  individuals  in  the  organization 
fail  to  change  their  beliefs  about  Quality  the  TQM  efforts  will  fail. 
This  is  the  situation  in  many  cases  in  organizations  starting  TQM 
where  management  dictates  a  change  in  behavior  for  the  individuals 
and  expect  TQM  to  then  just  happen.  What  is  needed  is  for  the 
performance  technologist  to  call  on  their  skills  to  develop  and  deliver 
a  training  program  that  can  assure  the  proper  program  outcomes. 
These  skills  include  ensuring  that  the  determining  of  instructional 
goals  are  shared  with  the  learner,  the  atmosphere  is  conducive  for 
learning,  the  learner  is  made  comfortable,  the  learners  experience  is 
valued  and  the  performance  technologist  takes  on  the  role  of  a 
facilitator  (Knowles,  1986)  (Rose,  1985).  These  things  can  ensure  the 
needed  training  support  is  in  place  to  implement  TQM. 

Training  and  education  are  major  parts  of  any  TQM  effort.  This 
training  is  for  understanding  the  TQM  process  and  the  training  that 
results  from  the  changes  caused  by  TQM.  The  training  referred  to 
here  is  that  training  required  to  understand  and  implement  TQM. 
Without  proper  training  any  effort  to  implement  TQM  is  destined  to 
failure.  Not  enough  emphasis  can  be  placed  on  the  need  to  train  all 
personnel  because  all  personnel  have  to  be  involved  in  any  TQM 
effort.  Not  only  do  the  leading  experts  emphasize  the  need  for 
continual  training  but  so  do  others  who  have  implemented  TQM,  such 
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as  the  USAF  Air  Logistic  Center  in  Sacramento.  There  they  have  set 
up  permanent  TQM  courses  for  all  employees.  Others  include  the  GM 
assembly  plant  in  Lakewood  ,  Georgia,  IBM,  Federal  Express  and 
Disney(Peters,  1987).  The  Federal  Government  has  set  up  the 
Federal  Quality  Institute  to  act  as  the  clearing  house  for  Federal 
Agencies  who  need  vendors  that  have  been  certified  to  provide  TQM 
training.  0PM  has  two  week  TQM  seminars  at  their  Federal  Executive 
Seminar  Centers  for  senior  Federal  Executives.  A  word  of  caution  is 
needed  because  mistakes  can  be  made  in  TQM  training.  Care  must 
be  taken  by  the  performance  technologist  to  avoid  these  mistakes. 
Some  of  the  common  mistakes  made  in  TQM  training  u  C 

Conducting  mass  training  for  everyone  before  support  systems 
such  as  steering,  process  teams  etc.  for  TQM  have  been  set  up. 

Overemphasizing  the  technical  tools  such  as  Flow  Charts, 
Histograms  at  the  expense  of  leadership  and  management 
issues. 

Oversimplifying  and  underestimating  the  difficulty  in  getting 
individuals  to  change  their  attitudes  about  Quality  and  TQM. 

The  undertaking  of  the  TQM  process  can  be  thought  of  as  a  three 
phase  effort.  The  first  phase  is  the  awareness  and  orientation  to 
TQM  models.  Dr.  Deming’s  model  advocates  the  extensive  use 
Statistical  Quality  Control  techniques.  This  includes  use  of  such  tools 
as  Pareto  Analysis,  Ishikawa  Fishbone  Diagrams,  Histograms, 
Control  Charts  and  Scatter  Plots.  Juran's  approach  is  a  model 
centered  around  three  major  quality  processes:  Quality  Control  and 
the  Quality  Sequence,  Quality  Improvement  and  the  Breakthrough 
Sequence,  and  Quality  Planning  and  the  Annual  Quality  Program. 
The  Crosby  approach  is  in  what  he  calls  the  Absolutes  of  Quality 
Management  and  the  Basics  Elements  of  Improvement. 

The  second  phase  is  the  introduction  to  the  tools  used  in  TQM.  These 
tools  are  as  provided  above.  All  advocates  of  TQM  rely  totally  on 
the  use  of  some  sorts  of  tools  to  collect  data.  Naturally  some  tools  fit 
more  appropriately  with  certain  models. 

The  third  phase  is  the  implementation  of  TQM.  Before  this  phase 
begins  a  decision  should  be  made  on  which  model  or  parts  of  models 
will  be  used  by  the  organization. 


The  roles  that  could  be  played  by  the  performance  technologist  will 
be  addressed  for  each  phase.  Before  doing  that  a  brief  description  of 
learning  outcomes  as  generally  accepted  by  many  performance 
technologists  is  needed.  The  five  kinds  of  learning  outcomes  (Gagne', 
1984,1985,  and  Gagne'  &  Briggs,  1979)  will  be  related  to  the  three 
phases  of  the  TQM  process.  These  outcomes  are:  intellectual  skills, 
verbal  information,  cognitive  strategies,  motor  skills  and  attitudes. 
The  matching  of  learning  outcomes  to  instructional  strategies  and 
task  performance  requirements  are  some  of  the  processes  that  the 
performance  technologists  normally  use  in  doing  their  work  as  well 

as  providing  instruction. 

The  first  phase  of  the  program  is  for  the  senior  management  and/or 
key  personnel.  Deming  believes  that  it  is  only  for  senior 

management.  Others  believe  it  is  for  the  explorers.  Hopefully  they 

are  the  same  persons.  The  performance  technologist's  role  at  this 

point  would  be  to  provide  briefings  to  those  persons  on  the  various 
concepts  of  the  Quality  Experts.  This  of  course  would  require 

research  by  the  performance  technologist.  It  might  also  require  that 
the  performance  technologist  receive  some  training  in  the  subject 
area.  Visiting  some  agencies  where  TQM  has  been  implemented 
would  also  be  helpful.  The  important  point  is  that  the  performance 
technologist  can  use  their  skills  and  knowledges  on  how  to  present 
verbal  information  and  concepts  to  decision  makers.  The  learning 
outcomes  would  be  as  follows: 

Intellectual  skills  consisting  of  concrete  concepts  and  defined 
concepts. 

Concrete  concepts  would  be  such  instances  as  identifying 
Demings  (14  )  fourteen  points  for  management. 

Defined  concepts  in  TQM  for  this  stage  would  be  the 
demonstration  of  the  use  of  TQM  in  an  organiza¬ 
tion. 

Verbal  Information  or  as  it  is  sometimes  called  "  declarative 
knowledge"  can  be  associated  with  the  learning  the  definitions 
of  such  things  as  TQM,  Executive  order  12637  etc. 

Attitudes  outcomes  are  very  important  at  this  phase  since  the 
learner  must  establish  new  beliefs  regarding  quality. 
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These  learning  outcomes  of  course  would  be  related  to  suitable 
instructional  strategies. 

The  second  phase  is  the  introduction  to  the  tools.  There  should  be  a 
broader  base  of  persons  involved  with  this  phase  than  those 
involved  in  the  first  phase.  Those  included  in  the  second  phase  are 
the  first  line  supervisor,  second  line  supervisor  or  manager,  team 
leaders  and  those  trained  during  phase  one  as  they  feel  necessary. 
At  this  point  the  performance  technologist  might  want  to  bring  in 
outside  support  for  those  areas  that  they  do  not  feel  comfortable  in 
presenting.  This  area  involves  using  some  statistical  models  as  well 
as  the  use  of  group  processes.  If  done  by  some  outside  agency  then 
the  performance  technologist’s  role  becomes  one  of  determining 
whether  the  presenter  can  perform  the  required  tasks.  The  learning 
outcomes  are  as  follows: 

Intellectual  skills  which  would  involve  the  learning  of  the 
concrete  concepts  of  the  tools  used  for  problem  identification 
and  analysis. 

Verbal  Information  as  indicated  above. 

Motor  Skills  outcomes  as  associated  with  the  use  of  the  various 
methods  of  collecting  and  munipulating  data. 

As  always  these  learning  outcomes  must  be  tied  to  the  appropriate 
instructional  strategies.  In  this  area  there  is  the  possibility  that  the 
presenter  of  the  material  could  be  an  outside  agency  and  that 
requires  the  performance  technologist  to  define  and  monitor  those 
activities. 

The  third  phase  is  the  implementation  phase.  At  the  outset  of  this 
ph  ase  management  should  have  determined  which  model  will  be 
used  to  implement  TQM  and  made  a  statement  of  management 
philosophy,  A  major  role  for  the  training  technologist  is  to  determine 
the  training  requirements  to  cause  the  cultural  changes  and  the 
required  paradigm  shift  and  to  help  people  through  the  land  of 
DABDA  (  Denial-Anger-Bargaining-Depression-Acceptance).  The 
learning  outcomes  for  this  training  are  as  follows: 

Intellectual  Skills  consisting  of  Defined  and  Concrete  Concepts 
and  Rules  and  Higher  Order  Rules. 
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Cognitive  Strategies 
Verbal  Information 
Motor  Skills 
Attitudes. 

The  performance  technologist  must  again  develop  the  appropriate 
instructional  strategy  for  the  particular  learning  outcome.  A  decision 
must  be  made  at  this  time  how  to  provide  the  continuous  training 
and  education  needed  to  support  TQM. 

Companies  that  are  successful  in  implementing  TQM  state  that  many 
years  are  required  to  fully  implement  TQM.  After  more  than  thirty 
years  it  is  said  by  the  Japanese  that  they  are  just  now  beginning  to 
fully  implement  TQM.  If  this  is  true  then  the  plan  for  training  must 
take  this  factor  into  account.  Long  term  continuous  training  is 
needed.  The  significant  point  here  is  that  there  is  a  very  important 
role  that  the  performance  technologist  must  play  in  the 
implementation  of  TQM.  TRAINING!!  As  stated  earlier  by  the 
experts,  training  is  a  very  vital  part  of  the  TQM  process.  Who  can 
better  research,  plan,  and  implement  a  training  program  for  TQM 
than  the  performance  technologist?  No  one! 

Another  area  where  the  skills  of  the  performance  technologist  are 
needed  is  in  the  assessment  of  performance  of  either  the  individual 
or  the  organization.  After  an  organization  decides  to  implement  TQM 
one  of  the  activities  carried  out  by  a  team  is  a  process  analyzation. 
This  includes  looking  at  a  process  to  determine  who  adds  value  to  the 
process.  This  effort  could  be  done  using  a  Flow  Chart  or  a  Fish  Bone 

Diagram.  It  is  important  during  this  assessment  that  discrepancies 

between  required  performance  and  actual  performance  of 
individuals  be  assessed  and  not  become  a  complicating  factor  in 
determining  the  direction  for  TQM  corrective  actions.  This  could 
easily  happen  if  the  team  doing  the  assessment  does  not  include 

members  who  possess  the  skills  that  a  performance  technologists 

have. 
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EVALUATING  THE  GENERAL  FEAS!B!U7Y  OF 
INTERACTIVE  COURSEWARE  AS  A  TRAINING  MEDIUM 

Jeri  Carter,  PhD.  and  Robert  D.  Perkins 


There  are  a  number  of  difficult  decisions  that  must  be  made  to  establish  and  manage  efficiently  and 
cost  effectively  an  Interactive  Courseware  (ICW)  development  project.  These  decisions  involve 
choosing  an  ICW  system  (including  the  hardware  configuration  arid  the  authoring  software)  and 
developing  a  staffing  plan  to  ensure  that  the  work  is  completed  on  schedule  within  the  project 
budget.  Before  any  of  these  decisions  can  be  addressed,  however,  the  primary  decision  must  be 
made;  Is  ICW  a  feasible  training  alternative  given  the  anticipated  training  situation?  The  purpose 
of  this  paper  is  to  provide  information  to  training  managers  about  a  structured  methodology  for 
making  the  initial  feasibility  decision.  The  general  feasibility  of  ICW  normally  is  evaluated  by 
examining  a  number  of  content,  student,  and  organizational  factors.  Subsequent  to  a  detailed 
discussion  of  these  factors  and  the  way  in  which  they  impact  the  feasibility  decision,  a  simpie.  user- 
friendly  decision  aid  is  presented  to  assist  training  managers  in  the  decision-making  processes 
related  to  ICW  feasibility.  An  informed  decision  at  this  stage  expedite?  -'^urr^Aare  development 
and,  therefore,  the  Instructional  System  Development  process. 


DR.  JERI  CARTER  Dr.  Carter  graduated  from  the  University  of  Oregon,  with  concurrent  advanced 
degrees:  a  PhD  in  Educational  Research  and  an  M.S.  in  Computer  Science;  Educational 
Applications.  After  completing  her  graduate  program  at  the  U  of  O,  she  went  to  work  for  Seville 
Training  Systems,  in  Dallas,  Texas.  Throughout  her  tenure  at  Seville,  she  worked  as  an  Instructional 
Research  Scientist.  Project  Director,  Technicai  Director,  and  Quaiity  Assurance  Manager  on  a  variety 
of  projects  in  both  military  and  corporate  environments.  Specific  duties  spanned  ail  phases  of  the 
ISO  process.  Currently,  Dr.  Carter  is  employed  by  Star  Mountain,  Inc.  as  the  Manager  of  San 
Antonio  Opt  retior  j. 

ROBERT  D.  PERKINS  A  graduate  of  Texas  A&l  University  (BS  Chemistry)  and  US  Naval 
Postgraduate  School  (MS  Operations  Research)  is  a  retired  US  Navy  officer  with  20  years  service 
in  submarine,  aviation  and  Acquisition  Program  billets.  He  is  currently  Manager  of  San  Antonio 
operations  for  D.P.  Associates.  Inc.  supporting  ATCfe  Training  Technology  Applications  Program. 


40 


EVALUATING  THE  GENERAL  FEASIBILITY 
OF  INTERACTIVE  COURSEWARE 
AS  A  TRAINING  MEDIUM 

Jeri  Carter.  PhD,  and  Robert  D.  Perkins 

INTRODUCTION  Presently,  Interactive  Course- 
\«are  (ICW),  a  term  used  in  the  Department  of 
Defense  (DoD)  to  refer  to  materials  needed  for 
interactive  training  delivered  via  a  computer 
(Joint  Service  Action  Group.  1989),  is  a  method 
of  instruction  highly  sought  in  military  training 
circles.  Preliminary  studies  indicate  that  ICW 
can  be  a  highly  effective  training  medium  In 
suitable  educational  environments,  it  has  been 
found  to  yield  significant  improvements  in 
trainee  achievement  and  transfer  of  training 
over  conventional  instructional  methods.  Some 
other  attractive  benefits  of  ICW  include: 

♦  Reduced  training  operation  and  maintenance 
costs 

♦  Improved  trainee  motivation 

♦  Automated  management  of  training  records 

♦  Unlimited  portability 

It  must  be  noted,  however,  that  the  key  phrase 
in  the  above  paragraph  is,  “suitable  educational 
environments."  In  spite  of  ICWis  relatively  rapid 
escalation  in  popularity,  due  largely  to  concur¬ 
rent  advancements  in  capabilities  and  reduc¬ 
tions  in  costs,  it  should  not  be  perceived  as  a 
panacea  for  all  inadequacies  in  existing  training 
programs  nor  as  the  undisputed  medium  of 
choice  for  new  training  endeavors.  Many 
decisions  to  incorporate  ICW  into  training  are 
made  solely  on  the  basis  of  a  collective  appetite 
for  high-tech  methods  and  machines  or  out  of 
a  competitive  propensity  to,  technologically 
speaking,  “keep  up  with  the  Jones'."  Only  an 
indepth,  systematic,  and  multi-dimensional 
assessment  of  curricula  and  its  goals  can  result 
in  an  accurate  determination  of  the  feasibility  of 
incorporating  ICW. 

PURPOSE  This  paper  provides  information  on 
3  structured  approach  to  determine  the  feasibili¬ 
ty  of  interactive  courseware  as  a  training  medi¬ 
um  in  a  given  training  situation.  Specifically,  a 
step-by-step  approach  to  the  decision  making 
process  and  a  detailed  description  of  the  com¬ 
ponents  of  this  process  is  delineated. 


THE  ICW  DECISION  The  general  feasibility  of 
ICW  is  evaluated  by  examining  a  number  of 
content,  student,  and  organizational  factors. 
These  factors  and  the  way  in  which  they  impact 
the  ICW  decision  are  discussed  in  the  following 
paragraphs. 

Content  Factors.  An  information  inventory 
administered  to  collect  data  in  support  of  this 
paper  showed  that  content  was  the  primary 
consideration  for  evaluating  the  feasibility  of 
incorporating  ICW  into  the  training  environment. 
The  following  content  factors  contribute  to  the 
feasibility  issue: 

♦  Content  characteristics 

♦  Content  stability 

♦  ICW  training  time 

♦  Course  life  cycle 

Content  Characteristics.  The  critical  content- 
related  element  is  the  appropriateness  of  the 
match  between  the  training  objectives  and  ICW 
instructional  features  and  other  capabilities. 
Because  ICW  is  a  flexible  medium  with  a  variety 
of  capabilities,  it  is  a  suitable  medium  for  pre¬ 
senting  many  knowledge-  and  performance- 
based  objectives.  Specifically,  ICW  is  especially 
appropriate  for  training  objectives  that  are 
dependent  upon  or  enhanced  by  graphic  repre¬ 
sentation,  photographs,  or  motion.  ICW  is  also 
well  used  for  instruction  requiring  group  prob¬ 
lem  solving.  Hewever,  it  is  contraindicated  for 
certain  levels  and/or  types  of  performance 
tasks.  For  example,  ICW  supplemented  by 
motion  sequences  is  an  excellent  choice  for 
demonstrating  interpersonal  skills.  The  facial 
expressions  and  body  language  can  be  exag¬ 
gerated  for  emphasis.  Conversely,  this  medium 
is  inappropriate  for  interpersonal  skill  task 
performance.  Current  technology  does  not 
allow  computers  to  be  programmed  to  evaluate 
this  type  of  performance.  Similarly,  training  a 
student  to  the  competent  and  highly  proficient 
level  of  task  performance  requires  hands-on 
experience  using  operational  equipment  or 
training  devices  (three-  dimensional  simulation, 
mock-ups,  fjart-task  trainers,  et  cetera). 

Content  Stability.  Content  stability  refers  to  the 
likelihood  of  the  course  material  to  change  over 
time.  Courseware  maintenance  refers  to  the 
process  used  to  modify  these  materials  be- 


cause  of  changes  in  operational  procedures, 
technical  manuals  or  publications,  or  the  equip¬ 
ment  that  is  the  focus  of  the  training.  Because 
content  revisions  require  accessing  and  repro¬ 
gramming  lesson  material,  ICW  is  doubtful 
when  courseware  is  expected  to  require  two  or 
more  major  content  revisions  per  year.  “Major" 
is  defined  by  the  percentage  of  course  material 
requiring  modification.  Specifically,  if  more  than 
20%  of  the  material  becomes  outdated,  the 
revision  is  "major." 

ICW  Training  Time.  ICW  training  time  refers  to 
the  number  of  ICW  lessons  in  the  course. 
Before  making  the  feasibility  determination,  an 
estimate  of  training  time  is  made.  It  is  rarely 
advisable  to  present  every  lesson  of  a  course 
through  one  medium.  The  content  character¬ 
istics  of  different  objectives  will,  almost  certain¬ 
ly,  require  a  media  mix  throughout  any  course. 
Prior  to  evaluating  the  content  for  "number  of 
lessons,"  the  nature  of  the  content  should  be 
examined  and  an  accurate  estimate  made  of 
the  number  of  ICW  lessons  in  the  course. 
Faulty  decision  making  could  result  if  the 
course  length  is  assumed  to  be  synonymous 
with  projected  number  of  ICW  lessons. 

Development  and  maintenance  of  ICW  is  expen¬ 
sive.  Large  portions  of  these  costs  are  accrued 
during  the  course  design  stage  of  the  Instruc¬ 
tional  System  Development  (ISD)  process. 
These  design  costs  are  partially  amortized 
across  lessons;  therefore,  the  cost  per  lesson  is 
inversely  related  to  the  number  of  ICW  lessons. 
Cost  also  is  impacted  by  learning  curve  time 
during  the  initial  stages  of  development.  Gener¬ 
ally,  courseware  developers  must  learn  to  use 
the  hardware  and  the  authoring  software,  and 
they  must  learn  to  understand  and  appropriately 
use  ICW  as  a  medium  of  instruction.  As  with 
design  costs,  the  impact  of  learning  cun.-s  costs 
decreases  as  the  number  of  lessons  increases. 

Recent  Air  Force  guidelines  for  selection  of  ICW 
as  the  medium  of  instruction  for  resident  train¬ 
ing  state  that  the  costs  (applied  to  ICW  equip¬ 
ment  acquisition,  lesson  development,  and 
courseware  maintenance)  must  be  offset  within 
two  years.  If  this  level  of  savings  cannot  be 
achieved  within  the  allotted  time,  ICW  should 
not  be  considered.  The  following  factors  are 
used  in  calculating  the  amount  of  savings: 


♦  Reduction  in  course  length 

♦  Reduction  of  operational  equipment  required 
and  the  associated  equipment  maintenance 
costs 

♦  Reduction  of  required  staffing  to  operate  the 
training  program  (the  savings  realized  by  a 
requirement  for  fewer  classroom  instructors 
must  be  balanced  by  ICW  development  and 
maintenance  requirements) 

Course  Life  Cycle.  As  with  number  of  ICW 
lessons,  life  cycle  of  the  course  also  impacts 
cost  inversely.  Development  costs  are  much 
lower  per  lesson  when  spread  across  a  long  life 
cycle.  Hewever,  one  complicating  factor  must 
also  be  considered.  The  longer  the  life  cycle, 
the  more  likely  it  is  that  major  revisions  will  be 
required.  An  accurate  cost  estimate  must 
reflect  maintenance  costs  as  well  as  develop¬ 
ment  costs. 

Student  Factors.  Student  factors  also  impact 
feasibility  of  incorporating  ICW  into  a  training 
situation.  The  projected  number  of  students 
and  the  characteristics  of  those  students  should 
be  examined.  As  in  the  previous  discussions 
on  training  time  and  course  life  cycle,  cost  per 
lesson  is  inversely  related  to  number  of  stu¬ 
dents.  Regarding  student  characteristics,  it  is 
difficult  to  examine  the  characteristics  of  future 
students.  Therefore,  it  is  necessary  to  review 
records  of  previous  students,  given  the  potential 
for  the  same  kind  of  students  to  matriculate  in 
the  future.  The  following  is  a  partial  list  of 
student  characteristics  that  support  use  of  ICW: 

«  Visually-oriented  learning  style 

♦  Adequate  reading  skills 

♦  Lew  motivation 

Given  use  of  the  full  range  of  ICW  capabilities, 
this  type  of  instruction  is  quite  helpful  to  stu¬ 
dents  who  are  visually-oriented.  Static  and/or 
animated  graphics  can  be  incorporated  to 
depict  or  illustrate  abstract  concepts;  IVD 
capabilities  can  be  used  to  demonstrate  proce¬ 
dures.  Even  highly  visual  lessons  probably  will 
use  some  text  to  identify  key  points.  Therefore, 
students  need  to  possess  "adequate"  reading 
skills. 

A  high  level  of  interactivity  should  be  designed 
into  lessons  to  enhance  the  learning  experience 
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of  poorly  motK/ated  students.  Other  techniques 
are  also  helpful  in  structuring  the  learning  of 
students  lacking  in  motivation.  Specifically, 
lessons  with  frequent  and  numerous  embedded 
questions  and  explicit  feedback  may  increase 
motivation  of  these  students.  Embedded  ques¬ 
tions  for  these  types  of  students  should  be 
designed  for  graduated  difficulty  so  that  a 
feeling  of  success  is  achieved. 

Organizational  Factors.  Another  set  of  factors 
that  impacts  the  feasibility  of  ICW  is  the  nature 
of  the  organization  into  which  the  training 
project  fits.  The  following  organizational  factors 
are  considered  in  determining  feasibility; 

♦  Current  training  equipment  configuration 

♦  Amount  of  organizational  support 

♦  Time  schedule  for  development 

Current  Training  Equipment  Configuration.  The 
current  training  equipment  configuration  is 
examined  to  assist  in  determining  feasibility. 
Environments  lacking  the  acquisition  budget  for 
hands-on  equipment  or  training  devices  for  task 
related  skill  development  training  could  fill  the 
gap  by  using  ICW  with  demonstration  and 
performance  simulation  capabilities. 

Amount  of  Organizational  Support.  Several 
kinds  of  organization  support  are  needed  to 
facilitate  the  decision  to  incorporate  ICW  The 
existence  of  this  support  should  be  examined 
prior  to  making  the  ICW  decision.  These  sup¬ 
port  factors  include: 

♦  Staff  availability  and  experience  with  ICW 

♦  Instructor  attitude  towards  ICW 

♦  Management  support  for  ICW 

In  evaluating  staff  resources,  examine  commit¬ 
ments  planned  for  the  time  frame  of  ICW  devel¬ 
opment.  Staff  availability  must  be  examined  in 
conjunction  with  other  commitments  and  the 
likelihood  of  dedicating  key  staff  members  to 
the  ICW  project.  Although  staff  experience  with 
ICW  is  not  prerequisite  for  implementation  of 
ICW,  experienced  staff  will  certainly  provide 
more  courseware  faster  and  cheaper.  Fre¬ 
quently,  instructors  are  initially  resistant  to  ICW. 
Negative  attitudes  can  hamper  development 
and  implementation.  Lack  of  management 
support  can  hamper  efforts  in  the  same  way. 


These  factors  need  to  be  examined  so  that  a 
judgement  can  be  made  about  the  relative 
amount  of  effort  required  to  fully  implement  ICW 
into  the  training  environment. 

Time  Schedule  for  Development.  In  some 
cases,  there  is  limited  time  available  for  devel¬ 
opment  before  students  are  scheduled  for 
training.  All  factors  evaluated  previously  may 
lend  strong  support  for  ICW,  but,  if  inadequate 
time  is  scheduled  for  development,  it  will  not  be 
a  feasible  choice.  In  evaluating  this  factor, 
remember  that  inexperienced  personnel  require 
more  development  time  than  experienced 
personnel.  In  addition,  instructor  resistance  to 
and  inadequate  management  support  for  ICW 
also  may  affect  the  organization^  ability  to 
meet  schedules. 

Decision  Factor  Summary.  Table  1  on  the 
following  page  summarizes  these  ICW  feasibility 
factors  and  their  impact  on  the  ICW  decision. 

ICW  Feasibility  Decision  Aid.  Decision  Aid  1 
has  been  developed  to  assist  in  evaluating 
feasibility.  This  Decision  Aid  (presented  after 
Table  1)  lists  the  following; 

♦  Each  factor  impacting  the  ICW  decision 

♦  A  yes/no  feasibility  question 

♦  Space  for  recording  your  answer 

Three  factors  on  this  decision  aid  are  especially 
critical  because  they  directly  or  indirectly  influ¬ 
ence  ICW  costs.  These  factors  include  content 
stability,  training  time,  and  course  life  cycle. 
"No"  responses  to  these  questions  provide 
strong  evidence  that  ICW  should  not  be  pur¬ 
sued.  These  three  factors  are  identified  on 
Decision  Aid  1  by  a  single  asterisk.  Another 
factor  (training  equipment)  is  also  critical.  A 
"yes”  response  to  this  question  strongly  indi¬ 
cates  a  need  for  ICW  This  factor  is  identified 
by  a  double  asterisk  on  Decision  Aid  1 .  When 
you  obtain  "no"  responses  to  the  cost-related 
factors  and  a  "yes"  response  to  the  training 
equipment  factor,  continue  with  the  decision¬ 
making  process.  Your  initial  cost  estimates 
may  be  in  error.  Probably,  the  cost  of  ICW  can 
be  justified  when  compared  with  the  cost  of 
sophisticated  training  devices  or  operational 
equipment.  When  you  obtain  "no"  responses  to 
all  four  of  these  factors,  you  have  little  justifica- 
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Factor  Type 

Factor 

Data  Examined 

Impact  on  ICW  Decision 

Content 

Content 

Characteristics 

Training 

objectives 

Consider  ICW  if  training  effectiveness  is  dependent 
on  or  enhanced  by  graphic,  motion,  or 
photographic  presentation. 

Content  Stability 

Likelihood  of 
major  revisions 

Consider  ICW  if  major  revisions  are  limited  to  one 
per  year 

Training  Time 

Projected  number 
of  ICW  lessons 

Consider  ICW  if  the  number  of  projected  ICW 
lessons  can  off-set  development  and  maintenance 
costs  within  two  years. 

Course  Life  Cycle 

Projected  course 
life 

Consider  ICW  if  the  course  life  expectancy  is  long 
enough  to  offset  a  reasonable  portion  of 
development/maintenance  dollars. 

Student 

Learning  Style 

Former  student 
records 

Consider  ICW  if  learning  style  is  highly  visual. 
(Resulting  ICW  should  also  be  visually-oriented  with 
heavy  use  of  graphics  and/or  IVD  capabilities.) 

Reading  Ability 

Former  student 
records 

Consider  ICW  only  if  reading  ability  supports 
projected  reading  level  of  anticipated  lessons. 

Motivation  Level 

Former  student 
records 

Consider  highly  interactive  ICW  with  frequent 
embedded  questions/extensive  feedback  if  low 
motivation  is  expected. 

Organizational 

Training  Equipment 

Availability  for 
hands-on  training 

Consider  ICW  with  simulation  capabilities  if 
hands-on  equipment  is  unavailable. 

Staff  Availability 

Resources  and 
commitments 

Consider  ICW  if  staff  is  available  for  dedicated 
assignment  to  ICW  project. 

Staff  Experience 

Staff  resumes 

Consider  ICW  regardless  of  staff  experience.  Plan 
for  learning  curve  time  if  staff  is  inexperienced. 

Attitude  toward  ICW 

Staff  meetings 

Consider  ICW  if  resistance  is  moderate  or  low.  Plan 
for  ICW  awareness  training  if  resistance  is  high. 

Support  for  ICW 

Management 

meetings 

Consider  ICW  if  support  is  evident.  Plan  for 
development  of  position  paper  if  support  is  low. 

Development 

Schedule 

Management 

meetings 

- - 1 

Given  a  short  schedule,  consider  ICW  if  staff 
experience  is  extensive,  instructor  resistance  is  low, 
and  management  support  is  high. 

Table  1.  ICW  Feasibility  Summary 
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Factor  Type 

Factor 

Feasibility  Question 

Yes 

No  1 

Content 

Content 

Characteristics 

Is  ICW  training  effectiveness  dependent  on  or 
enhanced  by  graphic,  motion,  or  photographic 
presentation? 

■ 

■ 

Content  Stability  * 

Are  major  ICW  revisions  limited  to  one  per  year? 

Training  Time  ’ 

Can  the  number  of  projected  ICW  lessons  offset 
development  and  maintenance  costs  within  two  years? 

■ 

■ 

Course  Life  Cycle  ’ 

Is  course  life  expectancy  long  enough  to  offset  a 
reasonable  portion  of  development  or  maintenance 
costs? 

■ 

■ 

Student 

Learning  Style 

Is  student  learning  style  highly  visual? 

Reading  Ability 

Does  reading  ability  support  projected  reading  level  of 
anticipated  lessons? 

■ 

■ 

Motivation  Level 

Is  low  motivation  expected? 

■ 

Organizational 

Training  Equipment  ’* 

Is  hands-on  equipment  unavailable  for  training? 

Staff  A/ailability 

Is  staff  available  for  dedicated  assignment  to  ICW? 

Staff  Experience 

Is  staff  experienced  in  ICW  development? 

■ 

Attitude  toward  ICW 

Is  resistance  to  ICW  moderate  or  low? 

■ 

Support  for  ICW 

Is  support  for  ICW  evident? 

Development 

Schedule 

Is  the  ICW  development  schedule  flexible? 

■ 

■ 

Decision  Aid  1.  iCW  Feasibility  Determination 


tion  for  ICW  Terminate  the  decision  process, 
and  pursue  use  of  other  media  for  training. 

Of  the  remaining  factors,  purcue  ICW  if  five  or 
more  "yes“  responses  are  obtained.  “Yes" 
responses  to  these  feasibility  questions  indicate 
general  feasibility  as  related  to  the  associated 
question. 

Exportability  Decision  Aid.  if  exportation  of 
ICW  to  remote  locations  is  planned,  additional 
feasibility  questions  must  be  answrered.  Use 
Decision  Aid  2  only  when  exportable  ICW  is 
planned.  Decision  Aid  2  is  presented  in  three 
parts  Answer  the  questions  on  Part  I  first. 


Exportability  Questions 

Yes 

No 

Is  there  existing  hardware  at 
the  field  unit  that  can  support 
ICW  training? 

Is  the  hardware  sufficient  to 
train  the  projected  number  of 
ICW  students? 

Is  the  existing  hardware 
available  for  ICW  training? 

Decision  Aid  2,  Part  I. 
Exportable  ICW  Feasibility 


If  you  obtain  three  “yes"  responses  to  the 
questions  on  Part  I,  skip  Part  II  and  proceed 
directly  to  Part  III.  However,  a  "no"  response  to 
any  one  of  the  questions  on  Part  I  requires  that 
Part  II  be  completed. 


Exportability  Questions 

Ye» 

No 

Is  budget  available  for 

hardware/software 

procurement? 

Is  budget  available  for 
operating  ICW? 

Decision  Aid  2,  Part  II. 
Exportable  ICW  Feasibility 


A  “no"  response  to  either  question  on  Part  II 
indicates  that  exportable  ICW  cannot  be  sup¬ 
ported  by  the  field  unit  budget  and  should  not 
be  attempted  until  funds  are  allocated.  "Yes" 
responses  to  Part  li  indicate  that  exportable 
ICW  is  feasible;  proceed  to  Part  III. 


Exportability  Questions 

Yes 

No 

Is  ICW  the  only  appropriate 
media  for  presenting  this 
content? 

Is  the  number  of  projected 
lessor.'  large  enough  to  justify 
ICW  use? 

Is  the  number  of  projected 
students  large  enough  to 
justify  ICW  use  ’ 

■ 

■ 

Decision  Aid  2,  Part  III 
Exportable  ICW  Feasibility 

“Yes"  responses  to  the  questions  on  Part  III 
indicate  that  exportable  ICW  is  feasib'e.  A  "yes" 
response  to  the  first  question  provides  evidence 
that  ICW  is  the  only  training  alternative:  this 
may  override  “no"  responses  to  the  second  and 
third  questions.  A  "no"  response  to  the  first 
question  on  this  part  and  a  "yes"  responses  to 
either  the  second  or  the  third  questrc.i  provides 
support  for  exportable  ICW 

CONCLUSION  This  paper  offers  a  structured 
approach  to  a  preliminary  training  requirements 
analysis  designed  to  determine  the  general 
feasibility  of  using  ICW.  The  methodology  of 
this  analysis  is  based  on  the  examination  of  the 
following  three  factors; 

♦  Content  of  the  training  program 

♦  Characteristics  of  the  prospective  students 

♦  Nature  of  the  organization 

Application  of  this  approach,  simplified  by  the 
use  of  decision  aids,  should  assist  in  preventing 
the  expenditure  of  excessive  time  and  financial 
resources  on  ICW  when  an  alternative  instruc¬ 
tional  medium  could  fulfill  the  requirements 
more  efficiently. 
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GUIDELINES  FOR  COMPUTER-BASED  TRAINING  (CBT) 
PLANNING,  SELECTION  AND  IMPLEMENTATION 


ABSTRACT 

This  project  forms  a  part  of  a  larger  piece  of  research  by  Armstrong  Laboratory,  Human 
Resources  Directorate  concerning  Computer-Based  Training.  For  some  time,  Armstrong 
Laboratory  has  recognized  the  need  for  standardization  of  CBT  planning,  selection,  and 
implementation  procedures  in  the  Air  Force.  This  task  was  undertaken  to  provide  research  into 
the  development  of  some  tools  which  might  be  used  to  address  the  needs  of  organizations 
planning  to  implement  CBT.  The  goal  of  the  research  is  to  improve  overall  training  within  the 
Air  Force,  and  in  particular  in  Air  Force  schools.  As  a  part  of  the  continuing  research,  the 
Laboratory  has  investigated  ways  of  improving  instructional  quality  by  providing  tools  and 
assistance  to  the  staff  of  Air  Force  training  centers.  This  project  makes  a  significant  contribution 
to  that  effort. 

The  specific  objective  of  this  research  was  to  determine  those  factors  which  are  critical  to 
Air  Force  organizations  which  are  planning,  selecting,  or  implementing  CBT.  By  determining 
critical  instructional  setting  variables,  an  organization  may  be  able  to  select  a  CBT  technology 
which  best  matches  its  requirements.  The  approach  taken  was  to  develop  a  questionnaire  and 
decision  aid  which  could  be  used  by  personnel  and  organizations  inexperienced  with  CBT  to 
categorize  their  training  needs  and  evaluate  CBT  technologies  in  light  of  these  needs.  Factors 
considered  are  the  various  types  of  cc  irse  objectives,  student  and  staff  needs,  equipment 
compatibility,  compatibility  with  other  training  organizations  and  courses,  cost,  and  several  other 
factors.  A  prototype  instrument  was  developed  which  was  aimed  at  performing  many  of  the 
same  functions  as  a  CBT  feasibility  study.  The  instrument  was  tested  on  three  typical  Air  Force 
organizations,  and  the  results  compared  to  the  recommendations  of  four  Air  Force  CBT 
consultants  who  were  convened  into  an  expert  panel.  The  instrument  was  revised  based  on  the 
results  of  try  outs  and  the  comments  of  the  CBT  expert  panel.  The  paper-based  instrument  is 
now  ready  for  operational  testing  with  user  commands,  and  eventual  autom.mon.  The 
instrument  provides  Air  Force  users  with  a  systematic,  organized  way  of  planning,  selecting,  and 
implementing  CBT  in  their  organization. 
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INTRODUCTION 

The  use  of  CBT  technologies  is 
widespread  throughout  the  Air  Force.  When 
CBT  has  been  properly  planned  for,  and  is 
implemented  based  on  a  sound  plan,  it  can 
have  a  positive  effect  on  both  training 
effectiveness  and  efficiency.  However, 
when  CBT  is  not  properly  planned  for, 
when  an  inappropriate  CBT  system  is 
selected,  or  when  a  CBT  implementation 
encounters  problems,  the  results  can  be  just 
the  opposite.  When  improperly  applied, 
CBT  can  potentially  result  in  ineffective 
instruction  resulting  in  substandard  learning, 
or  increased  costs  due  to  longer  training 
times,  courseware  development  problems, 
and  logistics  or  maintenance  problems.  In 
addition,  the  inappropriate  application  of 
CBT  can  result  in  adverse  impacts  on  a 
training  organization's  operating  structure, 
functioning  and  resources.  Given  the 
current  state  of  planning  for  and  selecting 
CBT  systems,  these  problems  do  not 
manifest  themselves  until  a  CBT  system  is 
being  developed  or  implemented. 

In  determining  the  appropriate  media 
application  for  their  training  environment. 
Air  Force  and  other  DoD  users  need  to  be 
able  not  only  to  select  CBT  from  a  group  of 
other  media  alternatives,  but  also  to  select 
the  most  appropriate  CBT  system  for  their 
specific  training  needs.  The  selection  of 
CBT  should  be  made  so  that  the  users  get 


the  most  powerful  system  for  training,  while 
the  Air  Force  has  the  benefit  of  a  cost 
effective  solution.  Since  the  Air  Force 
currently  has  no  standard  pitx:edures  in 
effect  to  determine  the  use  of  CBT  for 
instructional  applications,  the  way  in  which 
CBT  is  selected  and  procured  varies  firom 
command  to  command,  and  even  within 
commands.  Standardized  procedures  are 
needed  to  assist  Air  Force  and  DoD  agencies 
in  planning,  selecting  and  procuring  cost 
effective  CBT  systems  that  meet  their 
current  and  projected  training  requirements. 

Media  selection  models  abound. 
Certainly,  most  media  models  provide 
detailed  recommendations  as  to  the  most 
appropriate  media  to  use  in  accomplishing  a 
set  of  training  requirements.  However,  as  a 
group,  media  models  do  not  normally  assess 
an  organization's  ability  to  implement  a 
particular  medium.  Certainly,  none  make 
recommendations  as  to  how  to  implement 
the  instructional  technology  in  the  specific 
instance  of  the  organization.  Still  funher, 
little  is  available  to  help  assess  an 
organization's  ability  to  make  use  of  CBT 
technology.  What  this  report  summarizes  is 
the  results  of  a  study  undertaken  by 
Armstrong  Laboratory  to  develop  a  means 
of  advising  inexperienced  Air  Force  users 
on  whether  a  specific  CBT  technology  is 
appropriate  for  the  organization's  training 
needs,  and  to  provide  guidelines  which 
would  serve  to  hold  the  user's  hand  through 
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the  CBT  planning,  selection  and 
implementation  process. 

Originally,  the  research  task  focused 
on  determining  the  information  required  by 
an  organization  for  proper  CBT  planning, 
selection  and  implementation.  By 
identifying  this  information,  it  was  hoped 
that  some  linkage  could  be  established  with 
the  most  appropriate  CBT  technology, 
thereby  streamlining  the  CBT  selection 
process.  It  soon  became  obvious  that  the 
information  requirements  were  integrally 
linked  with  the  decisions  which  had  to  be 
made  by  training  planners  and  managers 
when  they  considered  the  introduction  of 
CBT  into  their  organizations.  At  that  point 
the  focus  shifted  to  the  CBT  decision 
process,  including  the  information  required 
to  make  it  work.  The  results  of  that 
investigation  are  reported  here.  Ultimately, 
the  goal  of  Armstrong  Laboratory  was  to 
develop  guidelines  to  aid  managers  and 
organizations  inexperienced  in  planning  for 
CBT.  Put  another  way,  the  primary 
objective  of  the  program  was  to  develop  a 
predictive  instrument  which  would  allow 
Air  Force  organizations  to  match  their 
training  requirements  and  instructional 
setting  environment  with  the  applicable 
CBT  system(s)  which  best  suits  them,  and  to 
assess  the  impact  of  such  technology  on  the 
organization. 

From  the  reactions  of  several  Air 
Force  organizations  which  have  been 
contacted,  CBT  planning,  selection  and 
implementation  assistance  is  long  overdue. 
Currently,  organizations  have  few  directives 
or  guidance  for  implementing  CBT.  The 
Instructional  Systems  Development  model 
may  work  well  in  other  areas,  but  when  it 
comes  to  implementing  CBT  the  model 
provides  little  specific  guidance  to  Air  Force 
users.  Furthermore,  within  the  Air  Force 
there  seems  to  be  no  model  to  follow  in 


planning  or  selecting  a  CBT  system.  Many 
Air  Force  organizations  must  either  obtain 
the  services  of  expert  CBT  consultants  to 
help  them  in  the  CBT  planning  process,  or 
rely  on  whatever  in-house  expertise  they 
may  have  available  to  do  the  task  for  them. 
In  both  cases,  there  might  be  different 
results  (given  the  same  data),  depending  on 
who  the  consultant(s)  or  in-house  experts 
are.  Research  has  indicated  that  a  lack  of 
clear  guidance  for  CBT  planning,  selection 
and  implementation  can  sometime  result  in 
hesitancy  on  the  part  of  the  organization  to 
implement  a  CBT  technology.  Because  of 
an  organization’s  uncertainty  as  to  what  and 
how  much  of  their  curriculum  can  benefit 
from  CBT,  there  may  be  a  tendency  to 
maintain  the  status  quo.  Even  worse,  some 
organizations  may  forge  ahead  blindly  into 
the  realm  of  CBT  without  the  necessary 
knowledge  of  how  CBT  may  impact  the 
training  requirements  or  the  organization  for 
better  or  for  worse. 

This  paper  describes  how  Armstrong 
Laboratory  has  set  about  solving  some  of 
the  CBT  planning,  selection  and 
implementation  problems  facing  the  Air 
Force  today.  It  will  describe  briefly  the 
methodology  employed  in  developing  a  set 
of  guidelines  to  assist  the  Air  Force  users 
through  these  difficult  problems.  It  will  also 
include  some  specific  features  of  the  CBT 
planning,  selection  and  implementation 
guidelines  which  resulted  from  the  study. 
The  instrument  itself  is  currently  available 
for  further  research  and  validation  through 
Armstrong  Laboratory,  Human  Resources 
Directorate  (AL/HRTC).  Although  the 
instrument  has  been  tested  in  the  field,  it 
cannot  be  considered  final  in  any  sense  of 
the  word,  since  there  still  remain  areas  of 
research  to  be  explored  in  CBT  planning, 
selection  and  implementation. 
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METHODOLOGY 

The  methodology  used  for  this  study 
consisted  of  the  following  major  steps,  each 
of  which  will  be  described  in  detail  in  this 
report: 

•  Determine  the  problem 

•  Develop  the  Instructional  Setting 
Inventory  (IS  I) 

•  Develop  the  CBT  Inventory 

•  Test  the  inventories 

•  Determine  CBT  decisions 

•  Validate  the  instrument  with  Air 
Force  users 

•  Validate  the  instrument  with  CBT 
experts 

Determine  the  Problem 

Initially,  the  researchers  focused  on 
potential  solutions,  i.e.,  an  updated  ISD 
model,  various  commercial  CBT  planning 
and  selection  programs,  etc.  TTie  ISD 
guidance  which  is  currently  available  to  Air 
Force  users  is  general  in  nature,  and  there 
seemed  to  be  no  good  reason  to  provide 
parallel  guidance.  Several  commercial 
products  available  provide  more  detailed 
advice,  but  none  of  them  is  tailored  to  the 
Air  Force  resident  training  environment. 
The  researchers  eventually  concluded  that 
generic  guidelines  would  only  be  of  minimal 
help  to  the  users.  While  such  documents  are 
useful  references  for  Air  Force  users,  only 
the  more  experienced  among  them  can  solve 
their  CBT  needs  using  such  products  alone. 
Certainly,  many  of  these  documents  go  a 
long  way  towards  expanding  the  knowledge 
base  of  the  users  regarding  CBT 
technologies.  But  none  of  them  are 
specifically  aimed  at  answering  the  difficult 
questions  which  face  an  organization  which 
is  contemplating  implementing  CBT.  Nor 
are  these  products  designed  with  the 


inexperienced  user  in  mind.  Many  of  them 
presuppose  that  the  user  knows  quite  a  bit 
about  CBT  technology  and  its  impact  on  the 
learning  enviroiunent  and  the  organization 
as  a  whole. 

The  researchers  focused  on  the 
question:  What  is  the  problem  in  Air  Force 
CBT  planning,  selection  and 
implementation?  They  relied  on  three 
sources  of  data  to  answer  this  question: 

1.  The  opinions  of  Air  Force 
personnel  who  had  recently 
implemented  a  CBT  program  at  their 
command.  This  helped  determine 
what  was  done  right  and  what  was 
done  wrong. 

2.  Intensive  interviews  with  Air 
Force  personnel  inexperienced  in 
CBT  planning,  selection  and 
implementation.  This  helped 
determine  what  the  inexperienced 
personnel  might  be  expected  to  know 
or  not  know,  and  what  that  person 
might  be  able  to  do. 

3.  The  researchers’  own  experience 
in  planning,  selecting  and 
implementing  CBT  for  Air  Force 
users.  This  formed  the  basis  for 
querying  the  users,  and  a  knowledge 
base  of  CBT  expertise. 

In  the  opinion  of  the  researchers, 
what  was  really  needed  was  to  mirror  the 
advice  that  a  skilled  consultant  would 
provide  to  an  organization  which  had  little 
or  no  CBT  technology  experience. 

Develop  Instructional  Setting  Inventory 

Looking  at  what  an  expert  CBT 
consultant  does  in  terms  of  an  Input- 
Process-Output  model,  the  research  task  is 
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twofold:  first  to  determine  what  data  or 
Input  the  consultant  uses  in  making  his 
recommendations  regarding  CBT,  and 
second,  to  model  the  Process  which  the 
consultant  goes  through  in  making  his 
recommendation.  The  desired  Output  was 
known,  or  at  least  the  form  that  the  users 
wanted,  namely  a  YesINo  recommendation 
regarding  CBT  and  some  advice  on  how  to 
handle  the  various  details  if  CBT  were 
selected.  Looking  at  the  problem  in  terms 
of  the  Input-Process-Output  model  means 
that  any  system  devised  to  help  the  typical 
Air  Force  user  has  to  gather  and  analyze  the 
same  kind  of  information  that  an  expert 
CBT  consultant  would  use  in  making  his 
recommendations.  The  logical  first  step 
seemed  to  be  to  determine  what  type  of 
information  an  expert  CBT  consultant 
needed  to  know  about  an  organization  and 
about  CBT  systems  in  general,  in  order  to 
make  his  recommendations. 

Several  personnel  on  the  staff  for 
this  project  had  performed  such  consulting 
work  with  Air  Force  user  organizations. 
These  personnel  were  quite  familiar  with  the 
types  of  questions  which  the  managers  of  a 
training  organization  usually  ask.  They 
were  also  familiar  with  many  of  the 
constraints  within  which  Air  Force 
organizations  had  work.  They  also  had  a 
clear  idea  of  what  kind  of  information  a 
training  organization  would  have  available 
or  be  able  to  get.  From  this  starting  point 
the  researchers  began  by  cataloging  the 
information  which  they,  as  CBT  consultants, 
would  need  in  order  to  advise  an 
organization  regarding  the  implementation 
of  CBT.  The  first  objective  in  determining 
the  information  requirements  for  CBT 
planning,  selection  and  implementation  was 
to  develop  an  Instructional  Setting  Inventory 
(ISI).  The  purpose  of  the  ISI  was  to 
characterize  Air  Force  resident  training 
school  environments  in  as  much  detail  as 


practical  and  usable.  The  intent  was  that  by 
using  the  ISI  one  could  encompass  the  entire 
range  of  training  center  and  school 
management,  administration,  training 
delivery,  courseware  development,  logistics 
support,  operations,  and  maintenance 
functions  performed.  The  hope  was  that  by 
being  able  to  fully  describe  the  resident 
training  school  environment  they  would  also 
be  able  to  identify  most  of  the  critical 
decisions,  key  factors,  variables, 
interactions,  and  information  requirements 
needed  to  fully  characterize  and  measure  the 
instructional  setting  for  the  introduction  of 
CBT.  Prior  experience  had  shown  that  each 
variable  in  an  instructional  setting  could  be 
critical  to  successful  CBT  implementation, 
therefore,  the  more  exhaustive  our  list,  the 
better  it  would  be. 

As  a  measurement  instrument,  the 
ISI  had  to  capture  data  on  an  organization  as 
it  currently  operates  and  does  business. 
Since  the  research  staff  included  personnel 
who  had  direct  experience  in  Air  Force 
training  center  operations,  it  was  relatively 
easy  to  identify  what  needed  to  be  asked 
about  current  operations.  Moreover,  the  ISI 
has  to  be  able  to  characterize  the 
organization’s  ability  to  adapt  itself  to  CBT. 
This  might  require  changes  in 
administration,  management,  logistics 
support,  or  other  organizational  factors 
besides  training  delivery  and  day-to-day 
operations.  So  in  addition  to  asking 
questions  about  what  is  currently  taking 
place,  the  ISI  has  to  be  able  to  capture  data 
which  could  be  used  for  making  predictions 
about  the  organization.  This  requirement 
necessitated  that  certain  questions  be 
inserted  into  the  ISI  which  gathered 
information  on  the  organization's  desired 
goals  and  future  intent.  This  was,  perhaps, 
the  hardest  part  in  constructing  the  ISI. 
Some  of  the  staff  felt  that  the  ISI  had  to 
approach  an  organization's  future  intentions 


without  biasing  the  respondents  in  favor  of 
CBT;  therefore,  it  shouldn't  ask  them 
anything  about  CBT  directly.  Ultimately, 
the  approach  taken  was  very 
straightforward.  The  IS  I  asked  the 
participants  about  the  expectations  they  had 
for  improving  their  organization  by 
introducing  CBT.  It  was  felt  that  this 
minimized  introducing  any  bias  in  favor  of 
CBT,  yet  it  still  got  the  required  information 
about  the  organization's  future  intentions. 

Develop  CBT  Inventory 

Just  as  the  ISI  was  developed  to 
characterize  the  Air  Force  training  center  or 
school  environment,  a  CBT  Inventory  was 
developed  to  allow  Air  Force  organizations 
to  categorize  the  functions,  allocations, 
capabilities  and  performance  of  currently 
available  CBT  systems.  Thus,  in 
combination,  the  ISI  and  the  CBT  Inventory 
would  allow  an  Air  Force  organization  to 
determine  their  organizational  climate  for 
CBT  and  to  match  their  requirements  with 
existing  CBT  system  capabilities.  The 
specifics  of  how  to  do  that,  namely  what 
decisions  regarding  the  organization  and 
CBT  had  to  be  made,  were  yet  to  be 
developed. 

There  were  differing  opinions  among 
the  research  staff  as  to  how  to  categorize 
CBT  systems.  Some  thought  it  might  be 
necessary  to  make  an  exhaustive  comparison 
across  vendor  CBT  systems.  This  database 
of  vendor  capabilities  might  parallel  the 
kind  of  information  that  CBT  expert 
consultants  would  have  available  to  them  in 
order  to  make  their  recommendations. 
However,  others  argued  that  it  would  not  be 
advantageous  for  the  CBT  Inventory  to  be  a 
catalog  of  current  CBT  systems.  If  the  CBT 
Inventory  were  to  take  that  form  it  would 
immediately  date  the  instrument.  If  a  single 
system  were  to  change,  the  CBT  Inventory's 


usefulness  would  be  diminished.  Rather,  the 
problem  was  approached  from  another 
angle:  How  could  an  organization’s  CBT 
reqmrements  be  described  so  that  the 
organization  could  effectively  evaluate 
various  CBT  technologies  available  to  meet 
these  requirements  at  any  point  in  time, 
current  or  future!  The  focus  of  the  CBT 
Inventory  was  changed  from  the  capabilities 
of  CBT  systems  to  defining  the  specific 
capabilities  needed  to  meet  the 
organization's  training  requirements.  This 
also  seemed  to  be  much  closer  to  what  an 
expert  CBT  consultant  would  do  in 
assessing  an  organization's  requirements, 
i.e.,  determine  the  requirements  and  then  use 
the  requirements  to  evaluate  available  CBT 
systems.  This  would  bring  into  play  the 
necessity  for  the  user  organization  to  make 
its  own  judgments  as  to  which  path  to  take. 
The  organization  would  be  able  to  judge  for 
itself  which  CBT  system  was  the  most 
advantageous  for  the  organization.  Cost  and 
other  factors  would  have  to  be  taken  into 
account  by  the  user  organization,  but  with 
some  assistance  as  to  how  to  weigh  each 
factor  in  its  decision. 

Test  Data  Collection  Instruments 

Once  the  two  inventories  had  been 
developed,  they  were  tested  with  members 
of  the  target  population  at  four  separate 
organizations.  The  pilot  test  was  performed 
to  determine  if  the  inventories  were 
collecting  the  right  data,  to  see  if  the 
required  data  was  readily  available  at  the 
participating  organizations,  and  to  ascertain 
that  the  questions  could  be  answered  by  the 
participants  without  the  assistance  of 
someone  outside  the  organization.  Since  the 
inventories  asked  some  detailed  questions 
about  various  components  of  the  curriculum 
and  the  organization,  it  was  determined  that 
it  might  be  necessary  for  some  participating 
organizations  to  have  several  different 


people  provide  the  needed  input  to  answer 
some  sections  of  the  inventories.  In 
addition,  the  involvement  of  more  than  a 
single  individual  in  the  CBT  planning  and 
selection  process  was  viewed  as  fostering  an 
environment  for  successful  implementation 
of  CBT  within  the  organization.  This  would 
allow  many  people  to  buy  into  the  CBT 
decision,  and  it  also  afforded  the 
opportunity  for  the  more  creative 
individual(s)  in  an  organization  to  emerge. 
These  personnel  have  proven  to  be 
invaluable  contributors  to  any  CBT  effort. 
The  pilot  testing  provided  us  some  useful 
information  on  both  the  content  and  format 
of  the  questions.  Some  of  our  test 
participants  who  had  some  CBT  experience 
even  went  so  far  as  to  provide  suggestions 
regarding  topics  they  thought  should  be 
included  which  had  previously  been 
omitted.  At  the  end  of  pilot  testing,  the 
instruments  were  revised  to  account  for  any 
problems  or  difficulties  which  were 
identified. 

Determine  CBT  Decisions 

It  soon  became  evident  that  the 
information  required  for  proper  CBT 
planning,  selection  and  implementation  was 
integrally  linked  to  the  decisions  which  had 
to  be  made.  At  the  start  of  the  project  the 
distinct  linkage  between  many  of  the 
decisions  was  less  obvious.  For  example, 
although  it  seemed  logical  to  separate  the 
decisions  regarding  hardware  and  authoring 
systems  from  decisions  regarding  authoring 
in  house  or  obtaining  contract  assistance,  it 
was  soon  determined  that  these  decisions 
were  quite  dependent  upon  each  other. 
Normally  authoring  system  decisions 
include  factors  regarding  ease  of  use.  These 
factors  are  linked  to  the  ability  of  the 
organization's  staff  to  author,  which  is 
linked  to  the  need  to  get  authoring  training 
for  the  staff,  which  is,  in  turn,  linked  to  staff 


turnover,  etc.  Each  decision  which  had  to 
be  made  seemed  to  have  some  kind  of 
impact  on  several  others.  Once  it  was  clear 
that  there  would  be  little  possibility  of 
separating  the  decision  process  cleanly  into 
two  parts,  the  idea  of  assessing  the 
organization's  setting  or  climate  for  CBT 
separate  from  the  kind  of  CBT  system 
which  would  be  suitable  for  it  was 
dismissed.  This  was  a  critical  conclusion 
for  the  format  of  the  eventual  product. 
Instead  of  two  separate  instruments;  one  to 
assess  the  instructional  setting,  and  another 
one  to  assess  the  capabilities  of  applicable 
CBT  technologies,  these  instruments  would 
have  to  be  combined  into  a  single 
instrument  which  would  provide  the 
required  data.  It  also  meant  that  the 
relationships  among  the  various  decisions 
would  have  to  be  determined,  and  these 
relationships  integrated  into  a  logical  CBT 
planning  and  selection  process  which  would 
allow  the  users  of  the  instrument  to  reach 
the  proper  conclusions  about  CBT  in  a  way 
that  was  transparent  to  the  user. 

The  approach  taken  in  determining 
the  decisions  which  needed  to  be  made  by 
an  organization  consisted  of  two  parts. 
First,  a  catalog  was  made  of  the  decisions 
which  had  to  be  made  in  planning  for  CBT, 
selecting  an  appropriate  CBT  system,  and  in 
implementing  CBT  in  an  organization. 
Included  in  this  catalog  were  the  decisions 
themselves,  the  decision  points  in  the  CBT 
planning,  selection  and  implementation 
process,  and  the  various  options  which 
should  be  considered  at  each  of  these  points. 
Project  researchers  reviewed  the  procedures 
they  had  successfully  used  before  in 
assisting  various  Air  Force  organizations  in 
selecting  CBT  systems  for  their  training 
requirements.  Each  researcher  listed  those 
decisions  and  decision  points  which  they 
had  encountered  in  the  CBT  planning 
process.  The  researchers  also  described  in 
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detail  the  information  needed  to  make  their 
recommendations.  This  step  resulted  in 
many  overlapping  and  complementary 
decisions  being  identified  by  the 
researchers.  Also,  much  of  the  information 
which  they  determined  to  be  useful,  was  in 
fact  needed  for  more  than  one  of  the 
decisions.  As  a  consequence,  before 
anything  further  could  be  done  to  structure 
these  decisions  into  some  kind  of  logical 
progression,  the  researchers  had  to 
determine  exactly  what  data  was  needed  for 
which  decision.  Then  they  had  to  sort 
through  the  decisions  which  had  been 
identified  to  determine  the  relationship(s) 
which  existed  among  them.  Of  necessity, 
this  second  step  involved  the  integration  of 
the  two  inventories  into  a  single  data 
gathering  instrument  which  could  be  used  in 
conjunction  with  the  various  decisions 
which  had  to  be  made. 


The  instrument  which  was  being 
developed  began  to  take  the  form  of  an 
Input  Component  (composed  of  the 
Instructional  Setting  and  CBT  Inventories, 
now  combined  into  a  single  Questionnaire), 
and  a  Process  Component  (consisting  of  24 
separate  yet  linked  decisions  through  which 
the  users  would  be  lead).  Figure  1 
summarizes  what  had  been  done  to  that 
point. 

The  final  step  taken  prior  to  testing 
the  instrument  on  a  user  organization,  was  to 
organize  the  decisions  into  a  logical  order. 
There  were  two  reasons  for  this.  First,  it 
would  let  the  users  know  where  they  were  at 
any  time  throughout  the  decision  process. 
The  often  misapplied  term,  user  friendly, 
was  the  watchword  in  developing  the 
document  User  acceptance  of  the 
instrument  was  critical,  and  it  was  felt  that 


the  more  user  friendly  the  document  was  the 
better  it  would  be  accepted.  If  the  research 
were  to  produce  a  document  which  would 
accurately  predict  an  organization's 
requirements  and  match  these  with  an 
appropriate  CBT  system,  such  an  instrument 
still  might  fail  unless  it  were  employed  by 
the  Air  Force  user  in  CBT  planning. 
Anything  which  would  contribute  to  user 
acceptance  of  the  instrument  was  evaluated 
and,  if  warranted,  included.  As  a  second 
reason,  the  structuring  of  the  decisions  could 
also  provide  another  organizing  principle  for 
the  items  in  the  Questionnaire.  (It  should  be 
pointed  out  that  the  primary  basis  for  the 
organization  of  the  questionnaire  was  the 
source  of  the  data.  For  instance,  questions 
about  information  which  would  most  likely 
be  found  in  the  same  documents,  office,  or 
area  were  grouped  together  in  the 
questionnaire.  This  was  thought  to  be  the 
simplest  way  to  structure  the  Questionnaire 
for  ease  of  use.) 

Another  word  about  user  friendly. 
This  paper  makes  a  point  that  the  instrument 
which  was  developed  for  Armstrong 
Laboratory  is  user  friendly  in  a  way  that 
none  of  the  other  documents  which  aid  CBT 
users  are.  What  is  meant  is:  when  the 
instrument  asks  questions,  they  should  be 
questions  the  user  is  able  to  answer,  when 
the  instrument  gives  directions,  the  user 
should  be  able  to  follow  them;  whenever  the 
instrument  directs  the  user  to  do  something, 
the  user  should  be  able  to  do  it;  when  the 
instrument  tells  the  user  that  he  will  be  able 
to  determine  something,  if  he  follows  the 
procedures,  then  it  should  be  obvious  to  the 
user  what  the  results  are,  and  when  he  has 
achieved  them,  etc.  As  an  example,  most 
CBT  decision  aids  ask  the  user  such 
questions  as:  Are  alternate  character  fonts 
needed?  In  our  opinion,  this  question 
should  never  be  asked  of  the  user.  The  user 
really  wants  and  needs  help  in  determining 


the  answer  to  this  question.  The  questions 
which  should  be  asked  are  ones  which 
determine  what  the  user's  requirements 
actually  are.  Such  questions  as  whether 
there  are  various  type  styles  used  in  existing 
materials  which  need  to  be  duplicated  in  the 
CBT  lessons,  etc.  The  procedures  which  the 
user  is  asked  to  follow  by  a  decision  aid 
should  tell  him  that  alternate  character  fonts 
are  or  are  not  a  factor  to  be  considered  in 
judging  which  authoring  system  to  use. 

What  does  the  structure  of  the 
decision  hierarchy  look  like?  Figure  2  is  the 
roadmap  of  decisions  extracted  directly 
from  the  instrument.  As  can  be  seen  from 
the  roadmap,  the  instrument  is  structured  so 
that  a  user  can  quickly  determine  whether 
some  CBT  technology  is  applicable  to  the 
organization's  training  requirements.  At  that 
point  in  the  process  the  user  reaches  a 
YesINo  decision.  If  the  organization  can 
make  use  of  CBT,  the  rest  of  the  instrument 
is  used  to  determine  how  to  best  implement 
the  technology.  If  the  initial  decision 
process  does  not  recommend  implementing 
CBT,  the  user  still  has  the  option  of 
completing  the  rest  of  the  decisions.  If  the 
user  decides  that  CBT  should  be 
implemented  even  though  it  has  not  been 
recommended,  he  must  still  complete  the 
rest  of  the  sections  of  the  instrument  to 
determine  the  specifics  of  implementation. 
As  he  works  his  way  through  the  various 
decision  points,  it  will  become  clearer  that 
he  has  made  the  wrong  decision.  Hopefully, 
the  users  will  come  to  the  realization  that 
CBT  is  not  always  the  most  appropriate 
technology  to  address  every  training  need. 

The  determination  of  cost  was 
purposely  left  to  last.  Several  of  the 
participants  in  the  test  group  commented 
that  they  appreciated  this  approach.  While 
each  organization's  decision  to  implement 
CBT  will  ultimately  be  decided  on  whether 
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the  time,  money  and  personnel  required  are 
available,  it  was  still  felt  that  each  decision 
regarding  the  applicability  of  CBT  should  be 
based  primarily  on  the  training 
requirements.  There  is  frequently  enough 
time  and  effort  devoted  to  determining 
whether  an  organization  can  afford  to 
implement  CBT. 

Validate  the  Instrument  with  Users 

In  order  to  validate  the  instrument, 
Armstrong  Laboratory  scientists  contacted 
various  Air  Force  organizations  which  were 
thought  to  be  in  the  process  of  planning  or 
selecting  CBT.  They  were  able  to  identify 
three  organizations  which  were  about  to 
begin  the  planning  process  at  that  time. 
These  organizations  represented  diverse 
organizational  types  (two  of  them  being 
formal  schools,  the  other  a  training  support 
organization),  with  different  course 
requirements  (one  course  involved  pilot 
training,  the  other  involved  avionics 
maintenance,  and  the  third  logistics),  and 
varying  levels  of  CBT  experience  (two 
organizations  were  inexperienced,  the  other 
had  a  few  personnel  with  CBT  experience 
on  staff).  Each  organization  was  contacted 
and  asked  to  participate  in  the  study.  Each 
was  enthusiastic  about  participating.  They 
felt  that  any  help  in  this  area  would  be 
useful. 

The  validation  process  consisted  of 
two  steps.  First,  the  organizations  were 
visited  by  the  Armstrong  Laboratory 
program  manager  who  provided  them  with 
copies  of  the  instrument  and  briefly 
consulted  with  them  on  how  to  make  use  of 
it.  The  program  manager  remained  on-site 
at  the  organization  throughout  a  portion  of 
the  time  that  they  were  using  the  instrument. 
This  was  done  to  ensure  that  the 
organizations  would  devote  sufficient  time 
to  working  with  the  instrument,  and  to 


answer  any  questions  which  might  arise 
regarding  interpretation  of  what  the 
instrument  meant.  The  program  manager 
was  fully  aware  of  what  the  instrument 
consisted  of,  but  he  did  not  participate  in  the 
development  of  the  questionnaire  or  the 
various  decisions  which  comprised  the 
instrument.  The  feeling  was  that  his 
presence  at  the  organization  would  ensure 
cooperation,  yet  still  allow  the  instrument  to 
stand  on  its  own. 

The  second  step  in  the  validation 
process  was  debriefing  the  organization  after 
they  used  the  instrument  to  determine  the 
applicability  of  CBT  technology.  A  staff 
member  was  sent  to  each  organization  after 
they  had  used  the  instrument.  The 
researcher  interviewed  each  participant  in 
the  program.  The  interviews  concentrated 
on  three  things: 

1.  Did  the  instrument  help  the 

organization  reach  a  decision 

regarding  CBT? 

2.  Were  the  users  satisfied  or 

dissatisfied  with  how  the  instrument 
helped  them  achieve  the 
organization’s  goals? 

3.  Was  the  instrument  user  jriendlyl 

Once  the  interviews  were  completed, 
the  information  gathered  from  each 

organization  was  compared  to  determine  if 
there  were  trends  in  their  comments.  There 
was  unanimity  among  those  participating 
that  the  instrument  helped  them  through  the 
CBT  decision  process.  If  they  were 
inexperienced,  they  said  that  the  document 
provided  organization  to  the  process  and 
guided  them  through  several  decisions 
which  they  would  never  have  thought  of 
otherwise.  If  they  had  some  experience, 
they  said  that  the  instrument  was  something 
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that  they  needed  two  years  ago.  Although 
there  were  several  comments  regarding 
desired  changes  to  specific  items  on  the 
questionnaire  or  formulas  in  the  various 
decision  sections,  each  participant  felt  that 
the  instrument  was  just  what  they  needed. 
Their  specific  comments  regarding 
suggested  changes  to  the  document  to  make 
it  more  user  friendly  were  also  collected  and 
analyzed.  The  changes  recommended  by  the 
participants  were  evaluated  with  other  data 
to  determine  which  ones  would  be 
implemented  prior  to  further  testing  of  the 
instrument  with  other  organizations. 

Validate  the  Instrument  with  CBT 
Experts 

Armstrong  Laboratory  was 
encouraged  by  the  results  of  the  pilot  testing 
of  the  instrument.  However,  a  second  test 
of  the  instrument  involving  CBT  experts 
yielded  even  more  surprising  results.  The 
Laboratoiy  has  always  had  concerns 
regarding  the  validity  of  the  instrument. 
Their  concerns  were  that  in  spite  of  the  fact 
that  the  instrument  was  successful  in  some 
individual  cases,  how  could  one  predict  that 
it  would  be  just  as  successful  in  other 
applications?  The  argument  was  posited 
that  the  instrument  was  supposed  to  take  the 
place  of  a  feasibility  study  which  would 
normally  be  conducted  by  an  expert  CBT 
consultant.  Therefore,  let  the  results 
obtained  from  the  instrument  be  compared 
to  the  results  of  an  expert  when  given  the 
same  information  about  the  organization. 

Armstrong  Laboratory  scientists 
agreed  that  the  best  way  to  assess  the 
validity  of  the  instrument  was  to  compare  its 
results  to  those  of  an  expert.  The  only 
question  which  remained  was:  Who  were 
the  experts?  The  Laboratory  contacted 
many  Air  Force  organizations  involved  in 
CBT  to  get  their  opinion  on  who  they  felt 


were  recognized  experts  in  CBT  technology. 
In  particular,  they  sought  experts  in  CBT 
planning,  selection  and  implementation. 
Several  experts  were  identified,  and  from 
that  group  four  were  selected  to  participate 
in  the  validation.  The  Laboratory  convened 
the  panel  of  expert  CBT  consultants  in 
October  1990.  TTie  group  of  experts  were 
highly  qualified  having  a  combined  total  of 
over  27  years  of  CBT  experience.  As  a 
group  they  had  been  involved  with  all  stages 
of  CBT,  in  particular  the  planning,  selection 
and  acquisition  of  several  large  scale  Air 
Force  CBT  systems  involving  hundreds  of 
users.  They  had  also  been  called  upon  from 
time-to-time  to  provide  advice  in  planning 
for  smaller,  more  specialized  CBT  systems. 
They  were  familiar  with  the  state-of-the-art 
in  CBT  technology.  Their  experience 
extended  to  numerous  different  hardware 
and  software  configmations.  The 
Laboratory  felt  confident  that  these  CBT 
experts  knew  what  they  were  talking  about. 

These  four  individuals  met  for  four 
days  to  validate  the  instrument.  The  experts 
were  only  partially  informed  of  the 
conditions  under  which  they  were  being 
assembled.  They  were  told  that  their 
services  would  be  needed  to  validate  a 
process  of  selecting  CBT.  When  they 
arrived,  they  were  briefed  by  the  Laboratory 
Program  Manager  regarding  the  general 
goals  of  the  program,  however  they  were 
not  given  any  information  about  the 
instrument  which  had  been  developed  nor 
the  results  of  the  first  pilot  test.  In  fact,  they 
were  unaware  that  the  instrument  had  been 
developed  yet. 

Having  thus  set  the  stage,  the  group 
was  given  its  first  task.  The  task  consisted 
of  asking  them  to  provide  expert  consultant 
services  to  an  organization  which  was 
considering  CBT.  One  of  the  two  pilot  test 
sites  was  selected  as  the  basis  for 
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comparison.  The  data  from  that  site  was 
made  available  to  the  expert  panel,  but  only 
ifrwhen  they  asked  for  it.  In  withholding 
the  information  that  had  been  gathered  from 
the  organization,  the  objective  was  to 
determine  if  the  experts  would  also  ask  for 
the  same  or  similar  information.  As  the 
information  was  requested  by  various  panel 
members,  it  was  provided  to  the  group.  The 
experts  were  observed  by  personnel  from 
Armstrong  Laboratory  and  from  the 
research  staff  to  document  whatever 
decisions,  recommendations,  and  questions 
they  might  have.  The  experts  were 
informed  that  they  would  have  to  debrief 
Laboratory  scientists  at  the  end  of  the 
second  day.  In  their  debriefing  they  were  to 
provide  whatever  advice  they  felt  the 
organization  would  need  in  planning, 
selecting  and  implementing  CBT.  They 
were  also  to  provide  a  description  of  what 
they  did  and  why  they  did  it. 

At  the  end  of  their  first  task,  the 
expert  panel  provided  their  CBT 
recommendations  for  the  selected 
organization.  The  recommendations  of  the 
expert  panel  and  the  conclusions  arrived  at 
by  inexperienced  personnel  using  the 
instrument  were  very  similar.  Both 
recommended  CBT  for  a  subset  of  the 
objectives;  both  recommended  using  a 
network  configuration  to  gather  CMI 
information;  both  recommended  using  CBT 
as  a  type  of  part-task  training-,  and,  both 
estimated  the  amount  of  time  and  personnel 
required  to  develop  CBT  in-house,  etc. 

During  and  after  their  briefing  the 
expert  panel  was  still  not  told  of  the  results 
arrived  at  by  inexperienced  personnel  using 
the  instrument.  All  of  their  input  was 
received,  essentially  without  comment 
regarding  its  relevance  to  the  validation  of 
the  instrument.  The  intention  with  regards 
to  the  expert  panel’s  recommendations  was 


to  use  these  comments  as  a  basis  for 
comparison  with  the  conclusions  arrived  at 
by  the  user  organization  using  the 
instrument.  But  the  results  achieved  were 
not  to  be  the  only  basis  for  comparison.  It 
was  also  considered  important  to  look  at  the 
other  two  components  involved  in  making  a 
CBT  decision,  namely  the  information 
required,  the  Input  and  the  method  used  to 
make  each  decision,  the  Process.  As  can  be 
seen  from  Figure  3  the  intention  was  not 
merely  to  match  the  recommendations  of  the 
panel  with  the  conclusions  of  the 
instrument,  the  Output.  That  would  still 
leave  a  black  box,  i.e.,  how  do  the  processes 
compare,  and  how  do  the  data  needed  by 
each  process  compare.  As  a  consequence, 
the  research  team  observed  the  procedures 
used  by  the  experts  in  arriving  at  their 
recommendation,  asked  them  to  explain  why 
they  did  various  things,  and  noted  the 
information  which  they  thought  necessary  to 
make  their  decisions. 

Once  they  had  completed  their 
debriefing,  the  expert  panel  was  informed  of 
their  second  task,  i.e.,  to  critique  the 
instrument.  They  were  informed  of  the 
purpose  of  the  instrument,  how  it  was 
created,  and  briefly,  how  it  was  organized. 
The  panel  members  were  directed  to  take 
the  document  away  with  them  to  review  on 
their  own.  Working  alone  they  were  to 
review  it  and  critique  its  contents,  format 
and  approach.  They  were  asked  to  be  as 
critical  as  they  could  be,  but  to  do  so  in  a 
positive  way,  i.e.,  whenever  there  was 
something  which  was  not  to  their  liking, 
they  were  asked  to  suggest  an  improvement. 
Each  expert  was  asked  to  prepare  another 
briefing  of  his  own  critique  of  the 
instrument.  Without  being  directed  to  do  so, 
each  person  focused  on  a  slightly  different 
aspect  of  the  instrument  during  his  critique. 
The  experts  provided  the  specific,  detailed 
criticism  expected  of  them.  On  the  whole, 
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their  acceptance  of  the  instrument  was 
remarkably  universal.  As  a  matter  of  fact, 
each  participant  requested  that  they  be 
provided  a  copy  of  the  finished  document  to 
help  them  in  future  CBT  planning.  One  of 
the  panel  members  suggested  that  a  follow¬ 
up  test  of  the  instrument  be  performed  with 
an  organization  which  he  knew  was 
contemplating  CBT,  but  was  inexperienced 
in  CBT  planning,  selection  and 
implementation. 

CONCLUSIONS 

What  the  Instrument  does  and  does  not  do! 

What  do  the  advice  or 
recommendations  provided  by  the 
instrument  look  like?  The  last  thing  which 
is  needed  or  wanted  by  user  organizations  is 
a  single  solution  to  multiple  problems.  In 
other  words,  a  standardized  CBT  system  for 
the  entire  Air  Force  (or  even  some  major 


commands  with  diverse  needs)  would  do 
little  to  provide  an  ‘appropriate  training 
solution  to  the  myriad  of  environments 

which  are  to  be  found  in  Air  Force  formal 
schools.  Also,  the  managers  of  training 
programs  do  not  need  a  tool  which  provides 
them  with  only  a  single  solution  to  match 
their  training  needs.  Managers  must  be  able 
to  make  decisions  given  various  alternative 
approaches  to  their  training  requirements. 
They  need  an  instrument  which  gives  them 
more  than  one  way  to  look  at  the  training 
requirements  and  potential  solutions.  As 
with  most  good  managers,  training 
managers  need  to  be  able  to  play  What  if? 
with  the  solutions  to  their  problems.  The 
instrument  which  has  been  developed  allows 
them  to  do  just  that. 

Since  the  advice  of  an  expert 
consultant  retained  for  a  feasibility  study  of 
this  type  would  normally  result  in  the 
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production  of  several  documents  needed  by 
the  organization  to  acquire  the 
recommended  CBT  system,  the  instrument 
attempts  to  emulate  that  same  approach. 
Ihe  Output  Component  of  the  instrument 
either  provides  or  causes  the  organization  to 
gather  the  necessary  information  required  to 
acquire  the  CBT  system.  An  expert  would 
probably  leave  an  organization  with 
something  approximating  the  information 
listed  below,  therefore,  the  instrument  also 
provides  the  same  data: 

1.  A  description  of  which  objectives 
can  and  cannot  be  converted  to  CBT. 
The  instrument  makes  this 
determination  right  up  front.  It  is  the 
basis  for  all  further  CBT  planning. 
The  instrum.ent  will  indicate  to  an 
organization  how  much  of  its 
curriculum  can  be  converted  to  CBT. 
It  leaves  the  decision  as  to  how  much 
will  be  converted  up  to  the 
organization’s  managers.  They  will 
have  to  determine  what  they  can 
afford  to  do  given  the  resources  that 
they  have  available  to  them. 

2.  Recommendations  as  to  which 
CBT  technology  (e.g.,  CAI,  IVD, 
DVI,  sirn’ilation,  etc.)  would  suit  the 
organization's  needs.  Such  a 
recommendation  would  normally 
consist  of  two  parts:  a  description  oi 
the  applicable  CBT  technologies  and 
which  objectives  the  technology  is 
suitable  for;  and  advice  concerning 
the  cost  effectiveness  of  the 
implementation  of  the  CBT 
technologies.  (The  instrument 
provides  specific  recommendations 
as  to  what  type  of  CBT  'chrology 
can  be  used.  The  instrument  also 
leads  the  user  organization  through 
its  curriculum  to  determine  how 
sophisticated  the  CBT  sysrem  and 


lessons  will  need  to  be.) 

3.  Some  rough  order  of  magnitude 
estimate  of  the  cost  of  implementing 
the  CBT  technology,  both  in  terms 
of  dollars  required,  personnel  needed 
and  facilities  change  requirements. 
(As  was  mentioned  before,  costs  are 
not  computed  until  the  very  end. 
When  costs  are  computed,  the 
organization  will  be  able  to 
determine  where  they  stand. 
Namely,  if  they  will  need  to  POM 
for  the  required  resources;  if  the 
amount  needed  is  beyond  all  hope  of 
ever  achieving;  or,  if  CBT 
technology  might  be  feasible  using 
in-house  resources,  etc.) 

4.  Tl»e  data  needed  for  a 
specification  which  the  organization 
could  use  to  acquire  the  system. 
Normally,  uo  expert  would 
recommend  a  single  specific  system 
as  the  only  way  for  an  organization 
to  go;  therefore,  tlie  specification 
would  be  based  on  tiie  organization's 
requirements  and  provide  the 
organization  the  means  of  judging 
a.mong  several  competing  systems. 
(The  instrument  does  not 
automatically  proouce  a 
specification,  but  it  does  gather  and 
organize  almost  all  ,.f  the 
information  required  to  construct 
such  a  specification.) 

What  does  the  instrument  not  do? 
This  document  was  rapidly  developed  and 
serves  as  a  prototype  for  CBT  planning, 
selection  and  implementation  decision 
making.  The  instrument  has  only  been 
tested  on  a  limited  number  of  organizations, 
therefore  it  must  still  be  considered 
preliminary  until  more  definitive  results  are 
obtained.  One  problem  associated  with 
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testing  this  instrument  is  that  the  number  of 
organisations  considering  implementing 
CBT  at  any  one  time  is  rather  small.  Once 
an  organization  contemplating  CBT  is 
identified,  getting  the  organization  to 
participate  in  the  study  will  probably  not  be 
a  problem  (each  organization  contacted 
during  this  study  readily  accepted  the 
opportunity).  Given  the  small  number  of 
organizations  contemplating  CBT  during  a 
specific  period,  obtaining  sufficient  data 
from  continued  testing  of  the  instrument 
might  pose  a  problem.  Depending  on  what 
criteria  were  established  for  acceptance  of 
the  results  of  such  a  test,  Armstrong 
Laboratory  may  be  able  to  continue  testing 
with  internal  Air  Force  resources. 

The  modular  structure  of  the 
instrument,  and  the  direct  linkage  which  has 
been  established  between  the  various 
questions  in  the  questionnaire  and  the  CBT 
decision  points  makes  the  instrument  a 
prime  candidate  for  automation.  Various 
levels  of  automation  can  and  should  be 
applied  to  the  instrument.  By  automating 
the  instrument,  its  usefulness  to  the  user 
increases  significantly.  As  a  paper-based 
tool,  the  instrument  is  limited  to  the  single 
set  of  answers  that  the  organization 
provides.  By  automating  the  system,  the 
instrument  allows  the  organization  to  try  out 
several  different  configurations  before 
settling  with  the  one  that  suits  their  needs 
and  they  can  afford.  Several  levels  of 
automation  are  available  to  the  Air  Force: 

1.  Simple  automation.  This  could 
be  accomplished  by  programming 
the  various  decisions  to  take 
advantage  of  a  database  of 
information  constructed  by  the 
questionnaire  and  objectives,  and 
some  kind  of  interactive  spreadsheet 
to  perform  the  cost  and  sizing 
functions. 


2.  Sophisticated  programming 
and  branching.  This  level  could  be 
achieved  by  more  extensive 
programming  to  link  various 
questions  and  change  parameters 
within  the  program  shell.  The  user 
would  be  branched  to  certain 
sections  or  around  others  depending 
on  answers  to  previous  questions. 

3.  Artificial  intelligence.  This 
would  allow  the  user  to  interact  with 
the  program  and  be  advised  about 
new  decisions  he  needs  to  make 
based  on  decisions  which  he  has 
made  p’^viously. 

Other  CBT  Decision  Aids 

Prior  to  developing  the  instrument, 
the  research  team  investigated  commercially 
available  CBT  decision  aids  which  might 
perform  comparable  functions.  Several 
systems  were  examined,  and  the  literature 
reviewed.  Although  there  are  several 
systems  available  which  perform  media 
selection,  and  others  which  perform  cost 
analysis  of  alternative  media,  none  of  the 
systems  examined  consider  the  complete 
range  of  variables  utilized  in  the  instrument 
to  make  a  recommendation.  It  was  not  the 
goal  of  the  researchers  to  conduct  a 
comprehensive  review  of  commercial 
automated  CBT  decision  aids,  but  they  did 
attempt  to  contact  a  number  of  vendors  to 
determine  what  kind  of  systems  were 
available.  After  reviewing  several  systems, 
the  researchers  determined  that  an 
inexperienced  Air  Force  organization 
considering  CBT  would  need  assistance  in 
several  more  areas  than  those  provided  by 
the  commercial  systems.  The  development 
of  the  instrument  was  aimed  at  plugging 
some  of  the  gaps  identified  in  these  systems. 
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OF  THE  BIGGEST  CCM^UTER  BASED 
TRAINING  SYSTEMS  IN  THE  WORLD 


Air  Training  Conmand  is  the  Executive  Agent  for  cryptologic  linguist  and 
analysis  and  reporting  training  and  the  Responsible  Training  Authority  for 
certain  general  intelligence  training  for  all  four  military  services,  as  well 
as  other  government  agencies.  All  Air  Force  intelligence  training  was 
consolidated  at  Goodfellow  Air  Force  Base,  San  Angelo,  Texas  in  1985.  Since 
collocating  this  training,  the  Goodfellow  Technical  Training  Center  (GTTC)  has 
been  in  the  forefront  of  enploying  state-of-the-art  corputer  based  training 
(CBT)  technology.  CBT  at  GTTC  is  being  irrplemented  through  the  acquisition  of 
the  SENTINEL  training  systons  (SENTINEL  ASPEN  and  SENTINEIL  BRIGHT)  .  These 
system.^  are  being  integrated  through  a  conceptual  program  called  SENTINEL 
CONCHO.  While  several  of  the  SENTINEL  training  systems  are  bigger  than  any 
other  training  system  in  the  Department  of  Defense  (DOD) ,  ccatibined,  they 
represent  the  largest  single  conputer  based  training  system  in  the  world, 
almost  1250  workstations.  The  two  phase  SENTINEL  ASPEN  program  trains  officer 
and  enlisted  personnel  in  general  intelligence  skills  including  imagery, 
targeting,  reporting  and  analysis,  and  briefing  skills.  The  two  phased 
SENTINEL  BRIGHT  program  trains  enlisted  personnel  in  cryptologic  skills, 
including  cryptologic  language  skills,  electronic  intelligence  special  signals, 
maintenance,  and  cryptologic  analysis  and  reporting.  This  presentation 
addresses  each  system  in  detail,  including  design  features,  operating  systems, 
authoring  languages  used,  and  in  particular,  unique  functionalities.  The  pros, 
cons,  and  achievements  of  each  system  will  also  be  discussed  with  eirphasis  on 
GTTC  efforts  to  develop  full  interoperability  among  systans.  Other  areas  of 
discussion  are  acconplishments  to  date  in  course  and  courseware  development, 
lessons  learned  in  the  realm  of  developing  courseware,  development  tools  and 
teams,  and  the  use  of  contractor  support  in  the  training  development  process. 

MR.  KENNETH  CASPER  Chief  of  the  Computer  Assisted  Instruction  Plans  Branch 
at  the  3480th  Technical  Training  Wing,  Goodfellow  Air  Force  Base,  Texas, 
received  a  Bachelor  of  Arts  degree  in  Russian  Area  Studies  from  Fordham 
University  in  1963,  and  a  Master  of  Science  degree  in  Education  from  the 
University  of  Southern  California  in  1970.  A  Colonel  in  the  Air  Force  Reserve, 
Mr.  Casper  served  over  11  years  on  active  duty  in  a  variety  of  intelligence 
assignments  in  Japan,  Vietnam,  and  Germany,  as  well  as  in  the  United  States. 
He  has  been  a  civilian  training  specialist  at  Goodfellow  since  1982,  holding 
the  positions  of  Training  Development  Branch  Chief  of  the  Analysis  and 
Reporting  Training  Division,  3480th  Technical  Training  Group;  Training  Advisor 
of  the  3495th  Technical  Training  Group;  and  Training  Manager  in  the  Collection 
and  Maintenance  Plans  Branch  of  the  3480th  Technical  Training  Wing.  He  has 
held  his  current  position  since  April,  1989. 


65 


O^^IE  OF  THE  LARGEST  CCWPUTER  BASED 
TRAINING  SYSTEMS  IN  THE  WORLD 

Mr.  Kenneth  Casper 


In  the  late  1970's,  at 
Goodfellow  AFB,  Texas,  a  CAI 
training  system  was  being  designed 
to  teach  cryptologic  linguists  the 
knowledge  and  performance  skills 
required  of  them  at  operational 
sites  all  over  the  world.  In  the 
traditional  linguist  training 
environment,  students  were  required 
to  listen  to  analog  audio  tapes  of 
foreign  languages,  write  short 
synopses  of  v^iat  they  heard  (write 
gists)  ,  and  also  write  corrplete 
verbatim  transcripts.  The  use  of 
thousands  of  tapes  on  individual 
tape  records  for  these  tasks  was 
both  time  consuming  and  distracting 
to  students  and  instructors  alike. 
Consider,  for  example,  the 
necessary  but  essentially 
unproductive  training  time  spent 
checking  out  and  inventorying  audio 
tapes,  searching  through  those 
tapes  for  the  appropriate  training 
material,  splicing  broken  tapes, 
and  maintaining  libraries  for  those 
tens  of  thousands  of  classified 
tapes.  The  method  was  time 
consuming,  clumsy,  distracting, 
fraught  with  security  weaknesses, 
and  inefficient.  There  had  to  be  a 
better  way. 

Thus  the  Voice  Processing 
Training  System  (VPTS) ,  also  known 
as  SENTINEL  BRIGHT  program,  was 
born.  Several  years  later,  its 
creators,  learning  quickly  from 
their  experiences  with  the  still 
embryonic  CBT  world,  developed  a 
second  phase  to  the  program  to 
teach  a  much  broader  range  of 
intelligence  and  intelligence 
related  technical  skills.  Thus, 
Goodfellow  developed  the  SENTINEL 
BRIGHT  I  and  SENTINEL  BRIGHT  II 
conputer  based  training  system  for 
technical  training. 


About  this  same  time  training 
managers  at  Lowry  Technical 
Training  Center  in  Denver, 
Colorado,  began  developing  a 
training  course  designed  to  teach 
imagery  specialists  of  all  four 
services  and  DOD,  modern 
state-of-the-art  digital  imagery 
techniques .  Incorporated  in  the 
new  course  was  corputer  aided  in¬ 
struction  (CAI) .  The  program, 
owing  to  its  place  of  birth,  was 
called  SENTINEL  ASPEN.  By  the  late 
1980' s,  after  realizing  the  ever 
growing  potential  for  computer 
aided  and  conputer  based  training, 
a  second  phase  was  also  added  to 
the  SENTINEL  ASPEN  program  as  well. 
Like  the  second  phase  of  SENTINEL 
BRIGHT,  this  follow-on  phase  was 
designed  to  train  more  generic 
general  intelligence  skills.  Thus, 
SENTINEL  ASPEN  I,  the  General 
Imagery  Intelligence  Training 
System  (GUTS) ,  and  SENTINEL  ASPEN 
II,  the  General  Applications  Intel¬ 
ligence  Training  System  (GAITS) , 
programs  were  also  established  at 
Goodfellow. 

These  two  SENTINEL  programs 
may  well  have  continued  in  parallel 
without  too  much  interaction,  had 
it  not  been  for  a  decision  made  in 
the  early  1980' s  by  the  Air  Force 
to  consolidate  all  Air  Training 
Command  (ATC)  intelligence  training 
at  Goodfellow  AFB.  The  result  was 
more  than  just  the  relatively 
simple  relocation  of  training 
courses  and  resources  fron  Lowry 
Air  Force  Base,  Colorado,  Offutt 
Air  Force  Base,  Nebraska,  and 
Keesler  Air  Force  Base,  Mississippi 
to  West  Texas.  There  were  really 
two  objectives  in  bringing  together 
all  intelligence  and  intelligence 
related  technical  training  to  one 
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ATC  center;  economics  was 
certainly  one  of  them.  This 
consolidation  would  not  only  be 
more  cost  effective  in  terms  of 
resources,  but  would  also  allow  for 
integration  of  the  training  itself. 
The  synergism  essential  to  the 
effective  use  of  intelligence  "in 
the  real  world"  had  long  been 
recognized  as  a  major  deficiency  in 
an  otherwise  effective  technical 
training  program.  (See  paper  on 
SENTINEL  CCXCHO  for  more  details.) 

As  a  further  natural  result  of 
this  intelligence  training 
consolidation  effort,  a  long, 
careful  look  was  taken  by  senior 
management  at  both  the  SENTINEL 
ASPEN  II  and  SENTINEL  BRIGHT  II 
programs.  Not  unexpectedly,  it  was 
discovered  that  they  had 
significant  commonality.  By  the 
late  1980' s,  the  SENTINEL  BRIGHT  II 
program  was  further  along  in 
development  than  its  SENTINEL  ASPEN 
counterpart,  so  vrf^en  it  came  time 
for  a  full  design  and  development 
effort  to  begin  on  SENTINEL  ASPEN 
II,  a  conscious  decision  was  made 
to  make  it  a  subset,  a  clone  if  you 
will,  of  SENTINEL  BRIGHT  II,  with 
options  for  further  functional  de¬ 
velopment  later  on  as  additional 
requirements  were  realized. 

Let  me  give  you  a  more 
detailed  description  of  the  four 
SENTINEL  programs  at  this  point  and 
then  discuss  the  larger 
inplications  of  their  being  collo¬ 
cated  at  Goodfellow  AFB. 

SENTINEL  BRIGHT  I,  as  already 
noted,  is  the  Voice  Processing 
Training  System  (VPTS)  .  It  is  the 
oldest  of  the  computer  assisted 
instruction  training  systems  at 
Goodfellow  and  is  used  to  train 
qualified  linguists  the  specialized 
vocabulary,  analysis  and  reporting 
skills  required  to  perform  their 
specialized  military  duties.  It 
was  designed  and  built  by 
Engineering  Research  Associates 
(ERA)  of  Vienna,  Virginia.  It  came 


on  line  in  1986  and  has  been 
training  intelligence  and  linguist 
specialties  since  then.  The  system 
is  ccitposed  of  460  workstations 
broken  down  as  follows:  385 

student,  60  stucfy  hall,  2  systems 
operator,  10  courseware  developer, 
2  training  administration,  and  1 
maintenance  workstations.  It 
should  be  noted  that  any  of  the 
student  or  study  hall  positions  can 
be  used  for  courseware  development, 
even  though  only  10  are  dedicated 
to  it.  The  entire  VPTS  is  served 
by  five  VAX  8600  mainframes 
pseudo-clustered  to  serve  a  varying 
number  of  classrooms  and 
workstations.  This  "host"  system 
was  designed  to  overcome  inherent 
single-point-of- fai lure 
disadvantages  of  a  wholly  clustered 
mainframe  syston.  If  a  single  VAX 
does  go  down  for  any  reason,  sore 
workstations  will  be  lost  and  sane 
functionality  of  remaining  worksta¬ 
tions  may  be  affected,  but  the 
entire  system  does  not  go  down. 
This  flexibility  is  an  important 
consideration  on  a  system  that  is 
used  on  a  two  shift  operation, 
training  students  who  are  attending 
courses  that  are  up  to  22  weeks  in 
length.  This  training  systan  is 
based  on  2  track  digital  audio  with 
CAI  and  CME  used  to  guide,  track, 
test,  and  remediate  student 
interaction.  Of  17,000  hours  of 
linguist  training  available  for 
conversion  to  corputer  based  tech¬ 
nology,  14,000  hours  were  selected 
for  actual  conversion.  12,000  of 
those  hours  have  already  been 
developed  both  by  government 
personnel  (civilian  and  military) 
as  well  as  civilian  contractors. 
The  EIBASMUS  proprietary  language 
developed  specifically  by  ERA  is 
used  exclusively  for  courseware 
development . 

SENTINEL  ASPEN  I,  is  also 
called  the  General  Imagery 
Intelligence  Training  System 
(GUTS) .  There  are  two  types  of 
training  workstations  used  in  the 
GUTS  systems,  hardcopy  and 


softcopy.  Both  systems  are 
downloaded  by  a  central  VAX  11/785 
mainframe. 

There  are  60  hardcopy 
workstations  each  of  which  consists 
of  traditional  light  tables  used  in 
conjunction  with  a  single  magazine 
quality  imagery  (M2I)  display,  a 
keyboard  and  a  mouse.  Imagery  and 
courseware  for  the  hardcopy  course 
are  downloaded  from  a  VAX  11/785 
mainframe  to  Micro  VAX  II 
mini-computer  clusters,  each 
serving  up  to  four  workstations. 
CA.I  functions  include  academic 
lesson  delivery,  testing,  and 
training  management  functions  such 
as  student  enrollment,  test 
administration,  and  student  records 
maintenance.  Courseware  is  being 
developed  in-house  by  government 
personnel  using  modified  ISS  (In¬ 
structional  Support  System) ,  a 
government  owned  programming 
language . 

There  are  16  softcopy 
workstations  (14  student,  1 
instiTJCtor,  and  1  authoring)  .  A 
softcopy  workstation  consists  of 
all  the  components  of  the  hardcopy 
workstation  plus  a  trac)cball,  a 
shaftbox,  and  dual  medium 
resolution  terminals  to  display 
digitized  M2I  and  courseware.  The 
additional  conponents  are  used  to 
display  and  manipulate  softcopy 
imagery  on  a  SM/COMTAL  processor 
which  replace  the  light  tables. 
Advanced  functions  to  manipulate 
the  imagery  (zoom,  warp,  mensurate, 
and  gray  scale)  are  incorporated  in 
a  sophisticated  software  package 
developed  under  contract.  The  SA  I 
program,  developed  under  contract 
by  the  Loral  Corporation  of 
Litchfield  Park,  Arizona,  also 
includes  80  hours  of  contractor 
developed  courseware  for  the 
softcopy  course.  Hardcopy  and 
softcopy  workstations  are  also  used 
by  authors  to  develop  and  deliver 
courseware  concerning  the 
exploitation  of  hardcopy  and 
softcopy  imagery  respectively. 


Like  the  VPTS,  GUTS  is  VAX  based, 
and  uses  the  VMS  operating  syst^. 

SENTINEL  BRIGHT  II  (SB  II), 
also  called  the  Cryptologic 
Intelligence  Training  System 
(CITS) ,  is  the  most  advanced  and 
ambitious  coiputer  based  training 
system  at  Goodfellow.  SB  II,  under 
development  and  delivery  contract 
by  American  Systems  Corporation 
(ASC)  of  Chantilly,  Virginia, 
employs  UNIX  based,  AT&T  6386 
stand-alone,  intelligent 
workstations  using  dual  screen 
technology.  Its  capabilities, 
which  far  exceed  the  capabilities 
of  any  of  our  other  conputer  based 
training  systems  at  Goodfellow, 
include  high  resolution  (1280  X 
1024)  touch  screen,  interactive 
video  disk  using  MacDonnell  Douglas 
laser  film,  and  X-Windows.  SB  II 
will  have  a  total  of  386  generic 
workstations  (some  with  specialized 
capabilities)  as  follows:  322 

student,  32  instructor,  21 
courseware  developer,  6  system 
operator,  and  6  training 
administrator  workstations .  They 
will  be  used  to  train  four 
different  intelligence  specialties 
including  advanced  linguist 
training,  cryptologic  intelligence 
analysis  and  reporting.  Electronic 
Intelligence  (ELINT),  and 
electronic  equipment  maintenance. 
All  courseware  is  being  developed 
by  government  personnel  using  the 
ACCORD  development  language.  It 
has  become  the  standard  for 
Goodfellow  CBT  because  of  its 
state-of-the-art  technology  and 
virtually  unlimited  potential.  It 
has  therefore  become  the  model  for 
SENTINEL  ASPEN  II. 

SENTINEL  ASPEN  II  (SA  II), 
also  known  as  the  General 
Applications  Intelligence  Training 
System  (GAITS) ,  is  our  newest  CBT 
program.  As  noted  above,  following 
the  standard  of  SB  II,  it  exploits 
commercial  off-the-shelf  technology 
using  high  resolution  color 
monitors  for  general  intelligence 
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training.  This  interactive  system 
program  has  275  workstations  as 
follows:  235  student,  18  instruc¬ 
tor,  14  courseware  developer,  4 
system  operator,  and  4  training  ad¬ 
ministrator  workstations.  SA  II 
will  focus  on  training  general 
intelligence  fundamentals  and 
applications,  targeting  and  weapons 
capabilities,  indications  and  warn¬ 
ing,  intelligence  correlation,  and 
Command,  Control  and  Communications 
Countermeasures  (C3CM)  courses .  In 
its  first  phase,  SA  II  is  a 
hardware  buy  that  is  fully 
coipatible  with  SB  II  and  includes 
provisions  for  purchasing  an  addi¬ 
tional  45  workstations  for  the  SB 
II  system.  Phase  two  of  SA  II  will 
add  specifically  developed 
functionalities  and  capabilities 
not  already  found  within  the 
SENTINEL  BRIGHT  II  software 
package.  Phase  2  will  not  begin 
until  sometime  next  year  (1992) . 

The  combination  of  SB  II 's 
original  386  workstations,  the 
additional  45  workstations  under 
the  SA  II  contract,  and  SA  II' s  275 
workstations,  which  I  call  sirtply 
SENTINEL  II,  constitutes  a  fully 
carpatible  and  interoperable  system 
of  706  workstations.  This  is 
clearly  our  largest  single  CBT 
capability  at  Goodfellow.  It's 
also  the  largest  coirputer  training 
system  within  ATC  and,  as  far  as  we 
know,  is  the  largest  single  CBT 
system  in  the  entire  Department  cf 
Defense . 

Having  three  or  four  large 
corputer  based  training  systens  at 
Goo^ellow  would  perhaps  not  be  of 
particular  noteworthiness  if  it 
weren' t  for  their  size  capabili¬ 
ties.  But  even  more  significant  is 
the  fact  that,  despite  having  two 
different  operating  systems  (VMS 
and  UNIX)  and  three  different 
development  languages  (ERASMUS, 
ISS,  and  ACCORD),  we  plan  to  tie 
them  all  together  into  one 


interoperable  CBT  system.  To 
further  ccnplicate  matters,  GTTC  is 
also  charged  with  creating 
exportable  training,  that  is, 
training  developed  at  the  school- 
house  for  independent  use  at 
non-ATC  units .  Because  we  are 
committed  to  satisfying  customer 
training  requirements  for 
exportable  training  using  their 
available  resources,  and  almost  all 
of  our  off-site  users  are  limited 
to  the  MS. DOS  environment,  we  are 
also  required  to  develop  modular 
courseware  in  the  MS. DOS  operating 
system.  For  this  reason  we  are 
doing  three  things  which  will 
greatly  expand  the  use  of  our 
training  products. 

First,  we  are  actively 
pursuing  the  development  of 
interfaces  and  other  software 
products  so  that  courseware  we  have 
already  developed  under  one  operat¬ 
ing  systen  is  readily  transportable 
to  other  operating  environments. 

Second,  we  are  actively 
pursuing  the  enhancement  and 
enrichment  of  the  MERLIN  authoring 
language  to  bring  it  up  to  the 
standards  and  capabilities  of  con- 
mercial,  proprietary  development 
languages  and  tools.  MERLIN  is  a 
fully  government  developed  and 
owned  courseware  development 
language.  Its  use  will  greatly 
simplify  our  courseware  development 
effort,  both  at  GTTC  and  in  the 
field. 

Third,  under  a  concept  program 
called  SENTINEL  CONCHO,  we  have 
designed  a  local  area  network  (I^AN) 
and  installed  a  dual  fiber  optic 
ring  at  Goodfellow  vhich  we  call 
the  Integrated  Intelligence 
Training  Support  System  (UTSS)  . 
It  not  only  makes  transfer  of 
information  and  courseware  from  one 
system  to  another  possible,  it  also 
allows  the  individual  training 
systems  to  receive  information  from 
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other  outside  sources  such  as 
national  intelligence  databases. 
As  a  result,  we  have  been  able  to 
establish  a  Training  Support  Data 
Base  (TSDB)  which  combines 
intelligence  source  data  and  other 
information  in  a  format  suited  to 
the  needs  of  our  courseware 
developers  and  instructors.  This 
TSDB  allows  the  different  training 
systems  and  courses  to  use  a  common 
database,  further  enhancing  the 
interchange  and  use  of  training 
materials  and  data.  Software  is 
currently  being  developed  to  allow 
the  ready  exchange  of  this 
courseware  without  the  requirement 
for  extensive  recompiling  or 
reconfiguration.  The  end  result 
will  be  considerable  savings  in 
courseware  development  time,  effort 
and  skills.  This  capability  for 
the  open  exchange  of  training  data 
and  materials  will  also  give 
Goodfellow  an  ability  to  modularize 
training  for  distance  learning  and 
exportable  training,  a  major 
objective  of  the  SENTINEL  CONCHO 
concept  of  operations.. 

The  creation  of  this  huge  CBT 
system  has  not  been  without  its 
challenges.  Essentially,  we  have 
faced  three  major  stumbling  blocks: 
courseware  development, 
interoperability,  and 
coirpatibility. 

There  is  little  I  can  add  to 
what  has  already  been  said  and 
written  in  these  conferences  and  at 
many  other  such  meetings  about 
courseware  development.  It  is 
expensive,  time  consuming, 
possessed  of  a  steep  learning 
curve,  requires  a  team  of 
dedicated,  well  trained 
specialists,  and  takes  a  change  of 
mind  set  on  the  part  of  those  doing 
it  and  those  receiving  it.  We 
have,  however,  made  what  we  believe 
to  be  significant  progress  in  all 
these  areas.  We  have  developed 
more  than  13,  000  hours  of 
interactive  courseware  on  three  of 
the  four  systems  (the  fourth  isn't 
delivered  yet ! ) . 


One  of  the  most  important 
lessons  we  have  learned  is  that  we, 
the  user,  really  do  know  more  about 
our  requirenents  and  capabilities 
than  anyone  else.  That's  not 
really  the  same  as  saying  that  we 
know  it  all.  In  fact,  we  hire  ex¬ 
perts  vihen  we  need  to,  because  they 
have  the  technical  knowledge  and 
skills  we  either  don't  have  enough 
of  or  don't  have  at  all.  We  use 
them  to  supplement  our  own 
capabilities,  not  to  si;pplant  them 
and  they  form  an  integral  part  of 
our  team  effort.  They  fill  the 
gaps  in  our  experience  pool  and  we 
assist  them  in  their  development 
tasking,  usually  as  si±>ject  matter 
and  training  experts.  In  that  way 
we  keep  open  the  lines  of 
coirmunication  and  ensure  that  long 
hours  of  courseware  development 
don't  go  down  the  wrong  path.  So 
far,  the  system  has  worked  well. 

We  have  also  developed 
training  courses  to  ensure  our  own 
home  grown  courseware  developers 
are  properly  trained  before  they 
get  thrown  into  that  briar  patch. 
In  addition  to  faculty  development 
courses  in  classroom  instruction 
skills,  test  and  measurenent,  and 
the  Instructional  System 
Development  (ISD)  process, 
government  courseware  developers 
take  a  6  week  course  in  CAI  devel¬ 
opment,  and  an  additional  week' s 
training  in  the  particular 
development  language  they  will  be 
using.  We  have  also  developed  a 
Style  (3uide  for  courseware  develop>- 
ment  and  have  periodic  cross-talk 
meeting  with  courseware  developers 
from  on-going  projects.  The  net 
result  is  a  well  trained  and  well 
informed  courseware  development 
team  concept  that  uses  standards 
specifically  designed  for  our 
training  requirement  and  system 
needs. 

Finally,  there  is  the  question 
of  expansion  and  ijpgrade.  SENTINEL 
BRIGHT  I,  the  Voice  Processing 
Training  Syston,  is  fully  meeting 


70 


our  current  training  requironents 
both  in  the  number  of  workstations 
and  in  the  functionality  of  the 
systsn.  Nevertheless,  the  system 
is  limited  and  we  are  looking 
actively  at  alternatives  for 
i^jgrading  and  ejqsanding  the  systen 
as  the  need  arises.  Since  the 
SENTINEL  BRIGHT  II  systan  (and  its 
clone,  SENTINEL  ASPEN  II)  are 
currently  at  the  leading  edge  of 
CBT  technology,  we  are  actively 
exploring  mutual  compatibility 
between  them  so  that  if  it  becones 
necessary  to  expand  the  SB  I 
system,  or  replace  parts  of  it  over 
the  foreseeable  future  we  can  do  so 
in  such  a  way  as  to  simultaneously 
expand  the  SENTINEIL  II  system.  We 
are  also  making  a  concerted, 
conscious  effort  to  insure  that  any 
futiire  systCTis  or  system  upgrades 
are  backward  compatible  so  that 
existing  courseware  will  not  be 
lost  or  require  redevelopment. 


In  summary,  four  major 
computer  based  training  systems 
have  been  established  at  Goodfellow 
Technical  Training  Center  in  San 
Angelo,  Texas  for  resident  training 
of  Department  of  Defense  military 
and  civilian  intelligence  special¬ 
ists.  These  individual  SENTINEL 
systems,  cortprising  a  total  of 
nearly  1250  interactive 
workstations,  are  being  joined  on  a 
fiber  optic  network  called  the  In¬ 
tegrated  Intelligence  Training 
Support  Syston.  A  vital  part  of 
this  overall  training  development 
and  delivery  program,  called 
SENTINEL  CONCHO,  is  the  Training 
Support  Database  being  developed 
for  the  common  use  by  students  and 
courseware  developers.  In  its 
totality,  the  integrated  computer 
based  training  program  at 
Goodfellow  Air  Force  Base 
constitutes  one  of  the  largest 
coiputer  based  training  systans  in 
the  world. 


SEOTINEL  CCM:H0 


Today's  challenges  in  intelligence  training  are  not  significantly  different 
from  the  challenges  in  other  technical  training  areas  throuchout  the  Air  Train¬ 
ing  Command  (ATC)  or  the  rest  of  the  Department  of  Defense  (DCD)  .  Except  per¬ 
haps  for  the  diversity,  coiplexity,  and  security  of  the  sources  of  the  informa¬ 
tion,  the  current  intelligence  training  environment  can  be  seen  as  a  microcosm 
of  the  full  range  of  challenges  being  faced  by  the  training  establishment 
throughout  government,  business  and  industry. 

Goodfellow  Technical  Training  Center  (GTTC)  has  developed  a  program  called 
SENTINEL  CONCHO  to  meet  these  challenges.  Its  goal  is  to  insure  more  and 
better  training  to  students  both  in  the  classrooms  and  on  the  job.  SENTINEL 
CONCHO  allows  us  to  cope  with  course  creation,  courseware  development,  and  re¬ 
source  management  in  an  integrated,  multi-source,  multi-level,  multi-technology 
environment.  The  program  has  three  phases:  (1)  local  integration  of  training 
and  training  systems,  (2)  distance  learning  and  exportabe  training  to  field  lo¬ 
cations,  and  (3)  an  integrated  exercise  network  with  outside  organizations 
world-wide.  The  pioneering  efforts  being  made  in  CBT  configuration,  interoper¬ 
ability,  and  integration  under  the  SENTINEL  CONCHO  program  will  have  a  signifi¬ 
cant  impact  on  technical  training  and  education  both  in  the  government  and  the 
civilian  sector  for  years  to  come. 

COLONEL  CHARLES  L.  ALDRICH  Conmander  of  the  3480th  Technical  Training 
Wing,  Goodfellow  Air  Force  Base,  Texas,  earned  a  Bachelor  of  Science  degree  in 
History  from  the  United  States  Air  Force  Academy  in  1969  and  holds  a  Master  of 
Arts  degree  in  Chinese  History  from  Indiana  University.  He  is  a  Distinguished 
Graduate  of  Squadron  Officers  School  and  has  conpleted  Air  Ccxiinand  and  Staff 
College,  the  National  Defense  University's  National  Security  Management  course, 
and  the  Air  War  College.  He  graduated  from  the  Naval  War  College  Senior  Course 
with  distinction  in  1987.  Colonel  Aldrich  was  trained  at  the  Armed  Forces  Air 
Intelligence  Center  at  Lowry  AFB  in  1970,  after  vhich  he  served  at  Tan  Son  Nhut 
AB,  Republic  of  Vietnam.  Other  intelligence  assignments  included  Langley  AFB, 
VA  and  Headquarters  Air  Force,  Washington,  D.C.  He  returned  in  1976  to  the  Air 
Force  Academy  where  he  served  as  Assistant  Professor  and  Course  Chairman  in  the 
Department  of  History  until  1980.  Later  assignments  included  the  North  Ameri¬ 
can  Aerospace  Defense  Command  (NORAD) ,  Air  Force  Space  Command,  Headquarters, 
United  States  European  Command,  Federal  Republic  of  (Germany,  the  Intelligence 
Community  Staff,  and  the  Air  Staff,  Washington,  D.C.  He  assumed  his  present 
position  in  July,  1991. 
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SENTINEL  CC»JCHO 


Colonel  Charles  L.  Aldrich 


In  the  early  1980' s,  the 
United  States  Air  Force  decided  to 
consolidate  all  Air  Training 
Camiand  (ATC)  intelligence  training 
at  Goodfellow  Technical  Training 
Center  (GTTC) ,  San  Angelo,  Texas . 
This  decision  was  significant  in 
several  respects.  It  recognized 
that  related  training  conducted  by 
geographically  separated  and 
independent  organizations  is 
inherently  unecononical,  resulting 
in  duplication  of  effort  in  course 
development  and  inconsistency  in 
training  delivery. 

Intelligence  training,  like 
training  in  other  highly  technical 
areas,  had  beccsne  fragmented  over 
the  years,  not  as  an  intended 
result  of  coordinated  decision 
making,  but  as  a  by-product  of 
isolated  decisions,  eadi  of  v^iich 
made  good  sense  in  itself,  but 
which  over  time  and  distance  lost 
their  potential  effectiveness.  By 
the  late  1970' s  the  inefficiency 
and  the  ineffectiveness  of  this 
fragmentation,  had  becone  apparent. 
Militciry  conmanders  in  operational 
decision  making  positions  were 
clearly  stating  that  fragmentation 
was  more  than  just  an 
inconvenience;  it  was  having  a 
detrimental  effect  on  the  quality 
of  training  their  intelligence 
officers  and  enlisted  specialists 
were  receiving.  Supervisors  in  the 
field  were  coipelled  to  spend  more 
of  their  very  limited  time  and 
resources  developing,  delivering 
and  administering  training,  and 
sometimes  retraining  technical 
school  graduates  in  basic  skills. 
Fragmentation  was  also  inhibiting 
the  long  term  goal  of  fielding  more 
rounded,  better  qualified  profes¬ 
sional  intelligence  personnel.  In¬ 
telligence  training  consolidation 
was  therefore  conceived  as  a  two 
step  process:  physical  collocation, 
and  then  integrating  the  training 


of  the  various  intelligence 
disciplines  themselves. 

The  first  step,  collocating 
existing  ATC  intelligence  technical 
training  courses  to  Goodfellow 
Technical  Training  Center  (GITC) , 
involved  moving  training  elements 
from  Keesler  AFB,  Mississippi, 
Offutt  AFB,  Nebraska,  and  Lowry 
AFB,  Colorado.  Doing  so  made  good 
economic  sense  since  the  classified 
nature  of  the  course  contents 
necessitated  the  establishment  of 
"secure"  training  facilities. 
Colloceting  the  courses  meant  that 
facilities  could  be  shared  and  that 
the  inevitable  duplication  of 
effort,  especially  in  the  area  of 
ccmtion  courseware  development  and 
training  acininistration,  could  be 
minimized,  or  perhaps  completely 
eliminated.  The  resulting 
collccation  and  consolidation  of 
management  resources  also  meant 
manpower  savings  and  a  more 
efficient  organizational  management 
structxore . 

As  noted  abo\'e,  the  decision 
to  consolicdate  was  not  solely  the 
result  of  economics .  There  had 
been  a  growing  perception  in  the 
intelligence  world  that  intelli¬ 
gence  training,  while  admittedly 
meeting  the  letter  of  carefully 
(defined  training  requironents,  was 
not  meeting  the  greater — largely 
implied — need  to  produce  well 
rounded  specialists  who  could  both 
see  and  understand  the  global  envi¬ 
ronment  of  their  skills.  The 
murmur  of  "gcxxl,  but  not  enou(^" 
wasn't  a  new  one,  nor,  in  fairness 
to  the  people  who  had  been 
prcxiucing  intelligenc:e  training  for 
many  years,  was  it  unicque  to  the 
intelligence  ccxntnunity.  The  same 
call  to  "give  us  more"  could  be 
heard  in  virtually  every  career 
field.  What  could  not  be  ignored 
was  the  equally  persistent  econcxdc 
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fact  that  training  time  cc^ld  not 
be  significantly  expanded  meet 
growing  training  needs  and  aesires. 

We  soon  realized  that  we  were  not 
being  asked  to  train  more  so  much 
as  we  were  being  asked  to  train 
better. 

As  it  turned  out,  the  decision 
to  consolidate  intelligence 
training  at  Goodfellow  came  at  an 
extremely  fertile  time.  The 
burgeoning  science  of  cor.  Jter 
technology  was  opening  new  vistas 
in  computer  capabilities, 
flexibility,  size  downscaling, 
memory  expansion,  and--most 
significantly — applications  soft¬ 
ware  in  fields  only  vaguely 
explored  in  the  past.  Although  it 
would  still  be  several  years  before 
the  full  potential  of  corrputers  was 
realized  in  the  training  world, 
Goodfellow  had  already  awakened  co 
this  potential  in  the  cryptologic 
linguist  training  area.  By  the 
late  1970' s,  Goodfellow  had 
designed  a  computer  assisted 
instruction  (CAI)  training  system, 
called  the  Voice  Processing 
Training  System  (VPTS)  .  This  was 
the  first  step  in  modernizing  all 
cryptologic  intelligence  training 
conducted  at  Goodfellow.  The 
overall  program  was  called  SENTINEL 
BRIGHT  and  the  VPTS  was  called 
SENTINEL  BRIGHT  I.  In  its  original 
form,  SENTINEL  BRIGHT  I  was 
designed  to  vpgrade  the  equipment 
and  equipment  handling  of  the 
language  training  orograms  at 
Goodfellow.  It  certainly  succeeded 
in  doing  that,  but  it  also  did  much 
more.  It  turned  out  to  be  only  the 
first  step  in  revolutionizing  the 
way  technical  training  itself  is 
designed,  developed  and  delivered, 
and  it  opened  the  door  to  three 
other  coipiiter  based  training  sys¬ 
tems  at  G^dfellow.  Over  the  next 
ten  years,  what  started  out  as 
little  more  than  tne  seed  of  an 
idea,  a  way  to  let  corputers  ndndle 
mechanical  and  trail  ling  management 
tasks  more  efficiently,  blossomed 
into  one  of  the  largest  computer 
based  training  systems  in  the 
world.  7 


The  collocation  of 
intelligence  training  courses  and 
resources  at  GTTC  was  ccnpleted  on 
schedule,  in  July,  1988.  It  was 
accomplished  without  a  single 
training  day  being  lost,  a  not  so 
insignificant  achievement,  vrtien  one 
considers  iJiat  nearly  one  hundred 
course.t  a^sd  almost  two  thousand 
students  a  day  were  involved.  The 
largest  share  of  these  transferred 
courses  and  students  came  from 
Lowry  AFB  in  Denver,  Colorado. 
With  them  also  came  another 
germinal  CBT  system,  an  imagery 
based  training  system  designed  to 
present  imagery  analysis  students 
training  using  film  (known  as 
hardc.  py) ,  and  conputer  generated 
imagery  (known  as  softcopy)  .  ITiis 
General  Imagery  Intelligence 
Training  System  (GUTS)  turned  out 
to  be  only  the  first  step  of  a  muon 
more  ambitious  program  called 
SENTINEL  ASPEN,  a  proqrani  ^signed 
to  modernize  all  non-cryptologic 
intelligence  skills  training  at 
GTTC. 

The  second  step  in  the 
intelligence  training  consolidation 
process,  and  by  far  the  tore 
ambitious  and  challenging  part  of 
it,  was  integrating  the  collocated 
intelligence  training  programs.  By 
1988  it  was  patently  clear  to  ev- 
-^ryone  that  the  heart  of  this  all 
.ncaipassing  and  intensive  training 
project  could  only  be  the  conputer. 
CXir  ultimate  goal  at  GTTC  was  to  go 
beyond  merely  teaching  and  testing 
the  finite,  relatively  easily 
measured  skills  and  knowledge  of 
esoteric  intelligence  specialties . 
Our  real  ambition  was  to  create  an 
iiicul]  inence  training  center  of 
excellence  that  revolved  around  a 
cohesive,  interlocking  computer 
based  training  environment.  We 
called  this  program  SENTINEL 
caoio. 

SENTINEL  CONCHO  is  more  than 
just  the  program  title  another 
interactive,  computer-based 
training  systea.  It  is  designed  to 
provide  a  coordinated  and  well 
4  integrated  intelligerice  training 


environment  for  apprentice  and 
advanced  level  stude  :s.  Its  goal 
is  more  than  just  bei  ter  individual 
training  courses  in  specialized 
areas.  The  new  training 
architecture  is  intended  to  create 
a  shared  learning  experience  by 
officer  and  enlisted  students,  a 
common  rs^ference  point  that 
culminates  in  professional 
interaction  among  several 
intelligence  disciplines. 

SENTINEL  CONCHO  is  being 
inplemented  in  three  phases.  These 
phases  are  basically  independent  of 
each  other  but  they  are  far  frcm 
being  exclusive  of  each  other. 
Elements  of  one  phase  become  part 
of  another  phase,  but  they  are  not 
sequential . 

Phase  I  is  Local  Integration. 

We  have  designed  and  installed 
a  local  cirea  network  (LAN)  called 
the  Integrated  Intelligence 
Training  Support  System  (IITSS)  . 
^  "'s  network,  which  is  catposed  of 
a  dual  fiber  optic  ring  and  a 
series  of  coaxial  subnetworks,  per¬ 
mits  the  exchange  of  data  among  our 
ft  X  host  CBT  systans  through  file 
transfer,  electronic  mail,  and 
virtual  terminal.  We  have  also 
created  a  TrainLng  Support  Database 
(TSDB)  which  will  be  used  as  a 
corrmon,  all-source,  integrated  in¬ 
telligence  information  "utabase  for 
all  cryptologic  and  general 
irt-lligence  training  development. 
Tile  long  term  econcmic  advantages 
of  this  design  are  obvious. 
Academic  courseware  developed  in 
one  course  on  one  system  can  be 
used  by  other  courses  on  the  same 
system  or  on  other  systems. 
Duplication  of  effort  is 
signficantly  diminished,  not  only 
in  the  initial  creation  of 
courseware,  but  also  in  updating  it 
because  ipdates  can  be  done  glo¬ 
bally  to  the  same  lessc.n  in  all 
courses.  P^ecause  the  soiurce  data 


for  all  courseware  develcpment  is 
the  same  for  all  courses,  the 
plague  of  inconsistency  in  data 
among  courses,  will  also  be 
overcome.  In  short,  all  students 
will  have  reference  to  the  same  big 
picture . 

As  noted  earlier,  the  SENTINEL 
CONCHO  concept  will  revolutionize 
technical  training  in  the 
intc!lligence  arena.  Basic  skills 
have  been  and  will  continue  to  be 
the  first  priority  of  technical 
training.  But  SENTINEL  CONCHO 
creates  a  truly  integrated  training 
environment  which  puts  these  skills 
in  context  frcan  the  first  day  of 
training  and  which  allows  the 
professional  intelligence  special¬ 
ist  to  grow  both  in  breadth  and 
depth  of  understanding. 

At  the  beginning  of  training, 
all  students  will  be  shown  the  same 
scenario  depicting  a  representative 
intelligence  situation  in  a 
realistic  intelligence  operations 
enviroiunent .  The  scenario  may  be  a 
reenactment  of  a  real  situation, 
e.g.  the  shootdown  of  KAL  flight 
007,  or  it  may  be  a  fictitious  (but 
realistic)  situation.  In  either 
case,  the  scenario  will  fccus  on 
the  actions,  reactions,  and 
interactions  of  intelligence 
specialists  across  the  entire  spec¬ 
trum  of  the  career  field  and  up  and 
down  the  career  ladder. 

Let  me  give  you  a  quick 
exanple. 

At  a  particular  military  site, 
the  commander  has  just  learned 
about  a  critical  situation 
developing  within  his  area  of 
responsibility.  As  the  conbat  con- 
mander,  he  requests  a  current 
intelligence  briefing  on  the 
situation.  We  see  a  team  of 
intelligence  specialists  do 
research  after  which  an 
intelligence  officer  (AFSC  8075) 


briefs  the  carniander  and  his  staff. 
Questions  are  asked  by  them.  The 
intelligence  officer  answers  those 
he  can  and  takes  chose  he  can't  as 
tasking  for  further  research.  At 
the  same  time  intelligence  reports 
are  being  received  from  various 
intelligence  sources.  The  picture 
shifts  to  the  sources  of  those 
reports.  We  see  particular, 
clearly  identified  intelligence 
specialists  collecting,  analyzing, 
collating  and  reporting  information 
and  we  see  how  the  reports  are 
handled  at  various  echelons.  The 
crisis  develops  and  we  see  the  vast 
array  of  actions  that  result.  Most 
importantly,  we  see  particular 
intelligence  specialists  actually 
doing  their  work. 

Students  will  see  how  their 
own  particular  specialties  function 
in  the  situation,  but  since  the 
scenario  includes  all  the  other 
intelligence  functions,  special¬ 
ties,  and  disciplines  which  can 
reasonably  be  expected  to  be  found 
in  such  an  environment,  the  student 
also  gets  to  see  how  each  of  them 
operates  in  coiimon  circumstances. 

The  student  will  then  receive 
che  specific,  criterion- referenced 
technical  training  required  for  his 
intelligence  specialty.  That 
content  of  technical  training  will 
not  be  much  different  from  what 
he /she  receives  now,  but  the  method 
of  delivery  will,  CBT  improves 
quality  of  training  by  making  it 
interactive  and  focusing  on  the 
student  rather  than  the  instnictor. 
Ideally  the  medium  of  training  is 
transparent  to  the  student.  In 
fact,  we  know  it  isn't.  Conputers 
will  at  least  make  it  more 
interesting  and  challenging  to  the 
student--a  good  motivator  in 
itself.  What  will  also  be 
different  from  the  training  of  the 
past  will  be  that  training  for  par¬ 
ticular  intelligence  specialties 
will  be  consistently  and  repeatedly 
referenced  back,  through  a  series 
of  "out-takes"  to  that  same  common 
scenario  that  all  the  students 


viewed  at  the  beginning  of  their 
training.  The  preponderance  of 
these  out-takes  will  be  to  satisfy 
specific  training  standard  objec¬ 
tives.  In  some  cases,  however, 
students  will  receive  non-testable 
orientation  and  education  on  other 
intelligence  specialties.  This 
will  give  them  a  more  objective 
view  of  vhere  their  specialties  fit 
within  an  intelligence  organiza¬ 
tion.  It  will  afford  all  students 
a  common  point  of  reference  in 
training;  it  will  eliminate  the 
feeling  that  they  operate  in  isola¬ 
tion;  and  hopefully  it  will  also 
motivate  them  to  learn  more  about 
their  own  specialties  and  about 
their  colleagues'  specialties. 

In  initial  skills  training 
courses  for  apprentice  intelligence 
specialists,  non-testable 
"out-takes"  will  naturally  be 
limited  in  number  and  scope.  For 
example,  linguists  might  be 
introduced  to  analysis  functions 
which  closely  parallel  their  own 
duties,  especially  in  areas  where 
they  would  be  expected  to  interface 
or  work  together  at  an  operational 
site,  but  they  would  not  receive 
very  much  exposure  to  imagery 
analysis  skills  or  duties. 

While  apprentice  level 
training  has  been  the  main  focus  of 
ATC  resident  training  in  the  past, 
and  will  remain  so  for  the 
foreseeable  future,  we  do  have  some 
advanced,  and  supplemental  training 
courses  at  Goodfellow.  Later  on  I 
will  discuss  distance  learning  and 
e.'-TX>rtable  training,  but  for  now 
let  me  just  say  that  the  advanced 
and  supplemental  intelligence 
courses  we  have  at  Goodfellow  will 
be  progressively  more  integrated. 
All  courses  will  retain  discipline 
dominance;  however,  skill -driven 
problems  will  be  based  on  the  inte- 
gration  of  multi-source 
intelligence,  and  include  the 
production  of  finished  intelligence 
products.  All  applicable 
supplemental-level  courses  will 
play  an  end-of-course  integrated 


exercise  in  a  scenario  generated  by 
the  tactical  simulation  program. 
In  this  way,  more  experienced 
intelligence  specialists  will  be 
exposed  to  other  intelligence 
related  specialties  vrtrLch  are  not 
as  readily  apparent.  For  exanple, 
an  imagery  analyst  may  learn  the 
rudimentary  principles  of  aircraft 
tracking,  or  a  targets  officer 
might  learn  more  about  the  sources 
of  data  from  which  targets  have 
been  selected. 

I  mentioned  tactical 
simulation  a  moment  ago.  Obvious¬ 
ly,  this  entire  program  is  heavily 
dependent  on  being  able  to  simulate 
actual  conditions  students  can 
expect  to  find  when  they  reach 
their  assignments.  Part  of  that 
simulation  capability  resides  in  a 
program  called  Tactical  Simulation 
or  TACSIM.  TACSIM  provides  the 
backbone  for  exercises  which 
simulate  real-world  operations 
through  scenarios  that  model  enemy 
(red  force)  and  friendly  (blue 
force)  positions  and  movements. 
Raw  intelligence  data,  that  is, 
unverified,  unevaluated  informa¬ 
tion,  and  finished  operational  and 
intelligence  reports,  that  is,  re¬ 
ports  of  verified,  or  evaluated 
data,  are  used  by  students  to 
assess  battlefield  situations.  The 
resulting  student  analysis  and 
reporting  directly  impacts  red 
force  posture  and  movOTent,  thereby 
providing  integrated  training  for 
all  intelligence  disciplines.  The 
TACSIM  program  allows  courseware 
authors  and  instructors  to  tailor 
scenarios  in  a  variety  of  condi¬ 
tions  and  circumstances.  These 
exercises  can  be  discipline 
specific  or  they  can  be  multi-dis¬ 
cipline  integrated  exercises.  Its 
tremendous  value,  of  course,  lies 
in  its  ability  to  allow  students  to 
interact  in  a  simulated  situation 
by  maneuvering  forces,  requesting 
additional  reconnaissance  reports 
and  air  strikes,  and  reviewing  and 
processing  collected  intelligence. 


At  the  end  of  all  courses, 
students  will  again  be  shown  the 
same  scenario  that  was  shown  at  the 
beginning  of  the  course  in  order  to 
reinforce  what  they  have  learned, 
give  them  a  better  appreciation  of 
the  other  disciplines  with  vdiich 
they  will  be  working,  and  show  than 
how  the  skills  they  have  learned 
fit  into  the  environment  in  viiich 
they  will  be  working. 

Phase  II  is  Exercise 
Integration , 

This  phase  is  conposed  of  two 
integrated  communications  networks: 

Intelligence  Training  Network: 
We've  seen  how  we  can  eliminate 
duplication  of  effort  in  training 
development  at  Goodfellow  by 
integrating  our  training  resources 
and  creating  a  Training  Support 
Database.  But  Goodfellow  isn't  the 
only  intelligence  training  center. 
The  Army  and  Navy  have  theirs,  as 
well  as  other  intelligence  and 
intelligence  related  agencies.  We 
don't  intend  to  pass  up  the 
opportunity  to  use  what  we  can  of 
their  training  development  efforts. 
We  also  want  to  share  with  them 
what  we  have  developed.  A 
communications  network  will 
therefore  be  created  between  the 
schoolhouse  and  other  DOD  training 
activities:  Ft  Huachuca,  AZ  (Army), 
Dam  Neck,  VA  (Navy /Marines) ,  and 
the  Defense  Intelligence  College 
(DIG)  in  Washington  D.C.,  among 
others,  in  order  to  further  the 
exchange  of  intelligence  training 
materials.  There's  also  another 
aspect  to  this.  With  this  kind  of 
connectivity  with  other  training 
centers,  we  can  also  participate  in 
joint  training  exercises.  A  link 
between  GTTC  and  the  Air  War 
College  at  Maxwell  AFB,  AL,  could 
also  promote  interaction  between 
Intelligence  amd  Operations  game 
planning.  These  exercises  will 
serve  two  very  important  but  up 
until  now  unattainable  objectives: 
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1)  Furnish  a  vehicle  for 
intelligence  personnel  to  exercise 
critical  intelligence  contingency 
skills  related  to  their  current 
positions. 

2)  Allow  intelligence 
personnel  to  participate  in 
appropriate  but  not  frequently 
experienced  scenarios  in  order  to 
expand  their  experience  base. 

Operations  Exercise  Network: 
A  cortmunications  network  will  be 
created  between  the  schoolhouse  and 
elements  of  the  intelligence 
community.  Interaction  will  be 
conprised  of  exercises  v^ich  may  be 
schoolhouse  driven  or  may  be  piggy¬ 
backed  off  exercises  driven  by 
other  agencies  (e.g.  the  Joint 
Chiefs  of  Staff) .  The  purpose  of 
this  interaction  is  two- fold: 

1)  Furnish  training  and 
experience  to  intelligence 
personnel  in  the  operational 
environment  in  aspects  of 
intelligence  which  their  everyday 
duties  would  not  normally  afford 
them.  This  will  maintain  and 
reinforce  the  "integrated"  nature 
of  their  awareness  and  make  them 
more  valuable  and  more  flexible  in¬ 
telligence  experts. 

2)  Provide  feedback  to 
the  schoolhouse  on  changes  taking 
place  within  the  intelligence 
community  that  will  give  it  an 
opportunity  to  update  and  upgrade 
its  in-house  and  exportable 
intelligence  training. 

Phase  III  is  Distance 
Learning. 

Resident  training  is 
expensive.  Its  cost  has  inhibited 
us  in  the  past  from  giving  more 
advanced  and  lateral  training 
courses  to  intelligence  special¬ 
ists,  especially  for  personnel  sta¬ 
tioned  overseas.  The  ccrobination 
of  teleconferencing  technology  and 
conputer-based  training  developed 


at  GTTC  offers  an  economically 
feasible  alternative  to  resident 
training.  Thanks  to  satellite  com¬ 
munications,  intelligence  personnel 
at  operational  units  will  be  able 
to  receive  fonnal  training  without 
liaving  to  leave  their  units.  Sane 
of  this  training  will  be  in  direct 
support  of  the  individual's 
supervisor  by  supplementing 
on-the-job  training  (OJT)  programs. 
Some  will  train  already  experienced 
intelligence  specialists  in  new 
systems  and  technologies;  others 
will  train  and  certify  personnel  in 
multiple  intelligence  skills. 
Clearly,  distance  learning  gives  us 
the  opportunity  to  expand  the 
breadth  and  depth  of  the 
intelligence  skills  pool.  In  addi¬ 
tion  to  technical  skills, 
exportable  training  affords  oppor¬ 
tunities  for  advanced  courses  that 
focus  less  on  technical  skills  and 
more  on  critical  thinking.  These 
will  include  mid-career 
intelligence  courses  for  officers, 
NCOS,  and  civilians  to  refresh, 
update,  and  expand  their  skills, 
and  ranotivate  them. 

Under  SENTINEL  CONCHO,  we  can 
now  train  intelligence  personnel  in 
an  environment  which  gives  them  not 
only  specialized  intelligence 
skills,  something  we  have  been  able 
to  do  for  many  years,  but  more 
iirportantly  a  new  awareness  and  ap¬ 
preciation  of  the  functions, 
duties,  and  responsibilities  of 
other  intelligence  functions.  For 
the  first  time,  we  will  be  able  to 
show  students  how  their  conbined 
skills  and  actions  contribute  to 
the  intelligence  mission  of  the  Air 
Force  and  the  other  military 
services,  and  how  they  individually 
and  jointly  form  a  real  community 
of  professionals  vrtio  have  a  very 
real  impact  on  the  national 
defense.  We  believe  that  throu^ 
the  SENTINEL  CCNCHO  concept  we  will 
be  able  to  foster  a  growing 
interaction  among  intelligence 
specialists.  We  see  in  the  future 
more  and  better  interaction  between 
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intelligence  officers  within  their 
own  disciplines  and  between  them, 
and  we  see  a  closer  working  rela¬ 
tionship  between  officers  and 
enlisted  personnel  at  every  level 
based  on  a  professional 
understanding  of  each  others 
talents  and  skills.  In  short,  we 
are  striving  to  produce  a  more 
cohesive,  interactive  relationship 
among  all  intelligence  disciplines. 

We're  certainly  not  there  yet. 
we  have  a  variety  of  goals  and 
tasks  to  accorrplish  over  the  next 
few  yeairs.  For  exanple,  we  have  to 
develop  scenarios  around  which 
specialty  training  can  be  focused. 
These  scenarios  must  be  credible 
and  representative  of  what  is 
currently  available  in  the 
operational  environment.  we  need 
to  identify  common  subject  areas 
among  all  intelligence  training 
courses  and  define  training  levels 
required  for  common  non-specific 
skill  training  (e.g.  security, 
safety)  for  students  within 
intelligence  courses.  And  we  must 
develop  and  iitplement  cannon  course 
materials  for  cannon  subject  areas. 

We  then  have  to  irrplanent  the 
IITSS  so  course  materials  developed 
for  one  system  can  be  transferred 
and  used  in  other  systems.  From 
that  combined,  unified, 
consolidated  coursewaire  database  we 
can  develop  exportable  (Type  6)  CBT 
training  materials  that  are 
compatible  with  field  site 
operational  and  training  equipment 
(hardware  independent) . 

We  also  have  some  long  term 
goals  that  are  intensively 
technology  dependent.  For  exanple, 
we  will  have  to  design  and 
implement  the  training  and  opera¬ 
tions  world-wide  networks  for  the 
interactive  exchange  of  training 


and  exercise  materials.  A  major 
stumbling  block  in  this  arena  will 
be  the  resolution  of  security 
access  problems  associated  with 
multiple  levels  of  world-wide,  in¬ 
teractive  intelligence  information. 
We  must  develop  courseware  and 
courseware  delivery  tools  that  make 
the  computer  based  integrated 
intelligence  training  system  user 
friendly.  But  perhaps  one  of  the 
most  inportant  things  we  have  to  do 
is  work  closely  with  our  custoners 
to  insure  they  are  fully  and 
properly  stating  their  training 
requiranents,  not  just  in  terms  of 
tedhnical  )<nowledge  and  skills,  but 
in  terms  of  the  global  environment. 

Finally,  computer  based 
training  is  not  sinply  a  better, 
more  efficient  way  of  doing  the 
same  old  job  of  training.  It  gives 
us  a  chance  to  go  far  beyond 
finite,  parochial  specialism.  CBT 
gives  us  opportunities  we  never  had 
before,  opportunities  for 
remediation,  for  flexible, 
objective  testing,  and  for  global 
environment  presentation  of 
specialist  and  technical  training. 

In  summary,  SENTINEL  COCHO  is 
the  vector  for  the  integration  of 
intelligence  training.  In  this 
capacity  it  establishes  training 
policy  and  the  content  of  the 
integrated  training  environment  it 
needs  to  produce  the  best  qualified 
apprentice  intelligence  specialists 
possible.  Specialists  will  be 
trained  in  the  shortest  possible 
time  to  perform  basic  intelligence 
skills  better  than  their 
predecessors  of  only  a  few  years 
ago.  More  than  that,  they  have  the 
tools  to  develop  into  the  most 
highly  qualified  professional  in- 
telligence  specialists  and 
technicians  ever  produced  by  Air 
Training  Command. 
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IN-HOUSE  COMPUTER  BASED  TRAINING  (CBT)  PROJECT 

FOR 

AIR  FORCE  ACQUISITION  SUPPORT  SYSTEMS  ENGINEERS 


Abstract.  There  is  a  significant  need  for  engineers  at  Aeronautical  Systems  Division  (ASD) 
who  understand  all  aspects  of  CBT  development.  A  CBT  project  was  initiated  in  order  to 
provide  an  effective  method  to  train  engineers.  Twelve  home  office  engineers  were  assigned 
the  responsibility  to  develop  the  following  courses:  CBT  Systems,  CBT  Request  for  Proposal 
(RFP),  CBT  Source  Selection,  and  CBT  Contract  Execution.  These  training  courses  have 
been  delivered  initially  to  at  least  twenty  direct  program  support  engineers.  This  paper 
discusses  the  project,  and  lessons  learned  for  similar  applications. 
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Introduction 

The  Computer-Based  Training  (CBT)  Project 
grew  out  of  the  increasing  scope  of  Air  Force 
training  system  acquisitions,  llie  Aeronautical 
Systems  Division  (ASD)  traditionally  procureo 
only  training  system  components  for  a  new 
weapon  system.  Now,  with  advances  in  training 
system  technology,  the  responsibility  for 
integration  of  the  training  system  is  shifting 
from  the  using  command  to  the  implementing 
command,  AFSC.  With  this  shift  has  come  new 
areas  of  concern  largely  associated  with 
courseware.  One  of  these  areas  of  concern  is 
CBT.  Engineers  have  had  to  adjust  to  this  shift 
and  get  their  feet  wet  in  the  ’’soft  science”  of 
courseware  design,  particularly  for  CBT. 
However,  a  vehicle  for  instruction  was  not 
readily  available.  The  Tactical  Air  Command 
(TAC)  Computer-Based  Interactive  Training 
System  (CBITS)  contract  c,  med  up  an  avenue. 

Project  Background.  The  Training  Systems 
Division,  ASD/ENET,  responsible  for  support 
of  any  proposed  training  system  acquisitions, 
recognized  an  opportunity  to  gain  needed 
project  experience  for  young  engineers  as  well  as 
develop  economically  the  training  courses 
necessary  to  prepare  support  systems  engineers 
for  the  courseware  aspects  of  their  jobs.  CBITS 
would  be  the  vehicle  for  training  development 
and  delivery.  The  Training  Systems  System 
Program  Office  (SPO),  ASD/YW,  agreed  to 
fund  and  procure  three  advanced  CBIT  Systems 
ex{7ecting  to  receive  the  benefit  of  the  courses  to 
be  developed.  TWelve  training  systems 
engineers,  military  and  civilian,  were  assigned 
(part  time)  to  the  CBT  project.  Initially,  these 
engineers  agreed  to  develop  four  courses  on 
CBT  acquisition:  CBT  Systems,  Request  for 
Proposal  (RFP),  Source  Selection,  and  Contract 
Execution.  The  CBT  Systems  course  was  to 
describe  the  hardware,  software,  and 
courseware  that  can  be  configured  for  a  CBT 
system.  The  sequential  RFP,  Source  Selection, 
and  Contract  Execution  courses  were  to  follow 
the  acquisition  of  CBT  from  the  establishment 
of  the  requirement  to  the  delivery  of  the  system 
to  the  ultimate  user.  The  target  audience  for 


these  courses  was  identified  as  training  project 
support  system  engineers  assigned  to  the 
Training  Systems  system  program  office. 

CBT  Project 

The  CBT  Project  was  initiated  in  June  1989. 
Engineering  approval  was  achieved  by  July, 
funding  in  August,  and  the  equipment  was  on 
contract  by  September.  The  CBT  Project 
schedule  is  shown  in  Figure  1.  The  project 
schedule  was  laid  on  top  of  the  existing 
workload.  A  program  manager  was  designated, 
but  access  to  project  personnel  was  through 
three  branch  chiefs.  The  Division  Chief  set  the 
project  priorities  and  received  periodic  progress 
reports  which  he,  in  turn,  reported  to  the 
Technical  Director.  Leadership  changes  did 
occur  during  the  project,  but  the  program 
manager  remained  constant.  The  program 
manager  directed  the  project  and  adjusted  to 
shifts  in  philosophy.  Project  personnel  were 
established  early,  but  several  were  lost  to  other 
programs  during  the  period  of  the  project.  The 
relative  inexperience  of  personnel  working  on 
the  project  meant  increased  time  for  learning 
efficiencies  and  understanding  how  to  use 
capabilities  such  as  the  graphics  editor  and  the 
training  management  function.  By  far  the  most 
difficult  problem  driving  the  schedule  was 
obtaining  agreement  and  approval  on  course 
content. 


Project  recommendation 

12Jun89 

Engineering  approval 

28  Jul  89 

Funding  approval 

14  Aug  89 

Equipment  on  contact 

26  Sep  89 

Authoring  software  training 

15  Oct  89 

Computer-based  instruction 

20  Jan  90 

training 

Preliminary  design  review 

16  Apr  90 

Equipment  delivered 

20  Apr  90 

System  design  review 

10  Jul  90 

Final  management/engineering 

26  Oct  90 

review 

Ready  for  Training 

1  Nov  90 

Figure  1 

ASD  Computer-Based  Training  (CBT)  Project 
Schedule 


The  initial  understanding  of  the  course  content 
was  so  general  that  when  the  engineers  began 
their  design  process,  boundaries  had  to  be  set. 
How  much  review  of  basic  acquisition  principles 
was  necessary?  What  was  the  process  for  a  CBT 
acquisition  as  opposed  to  a  training  system 
acquisition?  What  was  the  relationship  of 
courseware  reviews  to  traditional  hardware  and 
software  engineering  reviews?  In  the  process  of 
establishing  course  content,  an  acquisition 
process  for  CBT  courseware  was  derived.  See 
Figure  2.  Coming  to  agreement  was  time 
consuming,  but  brought  into  focus  where  the 
CBT  acquisition  process  needed  definition. 
Another  concern  was  how  to  communicate  this 
definition  to  the  engineers  implementing  the 
courseware  development  process.  Resolving  this 
concern  led  to  the  addition  of  three  other  lessons 
to  the  project:  Introduction,  Features,  and 
Technical  Overview. 

Courses.  The  resultant  set  of  courses  for  the 
CBT  Project  was  three  introductory  lessons  and 
four  courses  on  CBT  and  the  CBT  acquisition 
process.  The  Introduction  lesson  presents  CBT 
issues  in  the  systems  acquisition  process.  This 
lesson  assumes  the  engineer  already 
understands  the  systems  acquisition  process. 
The  Features  lesson  is  a  demonstration  of  the 
hardware,  software,  and  courseware  available 
for  CBT  along  with  the  developmental  features 
of  the  system  for  the  CBT  Project.  The  Technical 
Overview  lesson  provides  a  basic  overview  of 
CBT  and  the  relationship  of  CBT  within  a 
system  acquisition  at  ASD. 

The  four  primary  courses  begin  with  the  CBT 
Systems  course,  which  covers  the  purpose, 
operation,  and  distinct  features  of  CBT 
hardware  and  software.  The  next  course.  CBT 


Request  for  Proposal  (RFP),  identifies  the 
primary  issues  during  RFP  preparation  (limited 
to  CBT  acquisition).  The  Source  Selection 
course  focuses  on  the  technical  concerns  during 
the  selection  of  a  CBT  system.  The  Contract 
Execution  course  covers  the  areas  of  attention 
required  to  monitor  a  CBT  contracted  effort. 
These  courses  review  the  instructional  systems 
development  (ISD)  process  and  relate  the  ISD 
process  to  the  systems  engineering  process  for  a 
CBT  acquisition.  It  takes  approximately  8  hours 
to  accomplish  all  of  the  introductory  lessons  and 
primary  courses. 

Equipment.  The  CBT  system  purchased  for  the 
CBT  Project  was  based  on  the  CBITS 
configuration.  Adjustments  were  made  to 
enhance  the  system  with  the  most  current 
releases  of  hardware  and  software.  For  example, 
the  computer  was  upgraded  to  a  Zenith  386  (was 
Z-248),  the  Quest  authoring  system  was 
upgraded  to  version  3.0  (was  2.41).  Project 
personnel  had  the  opportunity  to  be  heavily 
involved  with  Quest  3.0  Beta  testing.  A  block 
diagram  of  the  CBT  Project  system  is  shown  in 
Figure  3.  A  brief  word  description  of  the  system 
capabilities  is  shown  in  Figure  4. 

Integration.  Integration  of  ail  necessary  CBITS 
components  plus  the  enhancements  caused 
difficulty.  The  estimated  amount  of  memory 
required  was  shortsighted.  The  VCR  was  not 
directly  controllable  from  Quest  software. 
Adding  the  laser  disk  along  with  the  VCR 
required  the  writing  of  indep>endent  drivers. 

Cost.  Three  CBT  systems  were  purchased  at  a 
cost  of  SlllK.  In  addition,  a  quality  television 
camera  system  with  editing  capability  was 
purchased  for  $24K. 
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Figure  2 

Acquisition  Process  for  Computer-Based  'TVaining  (CBT)  Courseware 
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Figure  3 

ASD  Computer-Based  Training  Project  System 


COMPUTER:  Z-386  with  a  25  MHZ  clock  DOS  3.3  + 

328  MB  Internal  Hard  Disk  386  to  the  Max  Memory  Mgr 

109  MB  Pluggable  Hard  Disk  Mouse 

3  1/2  inch  1.4  MB  Floppy  5  Slot  Expansion  Chassis 

5  1/4  inch  1.2  MB  Floppy  640K  +  1024K  R/'vM 

DISPLAY:  IBM  Info  Window  with  Touch  Screen  and  Voice  Synthesizer 

OTHER  Frame  Graffer  Color  Printer 

FEATURES:  Digital  Audio  Laser  Disk  Player 

Arts  and  Letters  Graphics  Editor  Image  Scanner 

Quest  Authoring  Software  Desgview 

Ethernet  Card  (Not  Connected)  T-EGA 

525  Line  Color  TV  Camera  System  PC  Paint 
with  Full  Edit  Capability 

Figure  4 

ASD  Computer-Based  Training  Project  System  Capabilities 
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Results/Lessons  Learned 

The  definition  of  an  acquisition  process  for  CBT 
courseware  (Figure  2)  was  a  milestone  for  this 
project.  The  meeting  of  the  minds  between 
team  members  from  multiple  technical 
disciplines  required  an  understanding  of  one 
another’s  vocabulary  and  recognition  of  the 
similarities  and  differences  in  how  to  approach 
the  same  problem.  The  real  melding  came  with 
the  integration  of  the  instructional  system 
development  process  with  the  systems 
engineering  process  description.  Now  the 
relationships  are  established  and  integrated 
product  development  is  facilitated. 

The  changes  in  leadership  over  the  course  of  this 
project  required  flexibility.  Each  change  shifted 
the  attention  of  team  members,  adjusted  the 
scope  of  the  project,  and  focused  on  concerns 
raised  that  now  needed  to  be  addressed.  The 
narrow  scope  on  just  CBT  acquisition  required 
a  venture  into  an  arena  not  experienced  by  most 
members  of  the  team.  Looking  back,  the 
definition  of  the  training  requirement  needed 
more  formalization  and  the  scope  of  the  work 
effort  suffered  from  changes  in  leadership. 

A  second  major  category  of  lessons-learned 
involves  the  need  for  a  firm  process  with  clear 
lines  of  authority.  This  is  especially  necessary 
when  a  CBT  project  is  done  ”ad  hoc,”  with 
part-time  help  from  many  organizations.  Just 
like  a  CBT  project  done  by  industry,  we 
wrestled  with  firm  spiecs,  milestones,  reviews, 
integration,  and  perhaps  most  importantly, 
content  integration. 

The  opportunity  of  carrying  out  an  in-house 
project  gave  the  young  engineers  on  the  team 
a  real  program  experience.  The  leaming-by- 
doing  drew  out  lessons  learned  that  could  never 
be  gained  in  the  classroom.  The  whole  process 
of  getting  the  CBT  system  operational,  learning 
how  to  use  it,  and  then  independently  developing 
courses  which  would  integrate  into  a  composite 
set  was  a  challenge  in  and  of  itself.  Then,  living 
under  the  pressure  of  time  to  delivery  and  at  the 
same  time  working  under  the  scrutiny  of 
multiple  levels  of  review,  feedback,  correction. 


then  repeat  the  cycle,  brought  a  feeling  of  reality; 
this  is  what  it  is  like  to  meet  a  program  deadline. 
The  joy  of  an  on-time  delivery  had  real 
significance. 

The  ability  to  carry  out  an  in-house  project  with 
limited  resources  on  top  of  existing 
commitments  and  any  new  commitments  taken 
on  during  the  course  of  the  project  was  an  extra 
challenge  for  the  program  manager.  The  team 
effort  required  the  support  of  the  formal 
organization  to  establish  the  right  priorities  and 
then  make  the  commitment  to  follow  through 
with  the  project.  With  the  formal  organization 
enforcement,  the  program  manager  had  the 
authority  he  needed  to  carry  the  project  to 
conclusion. 

The  good  start  of  CBT  instruction  on  developing 
courseware  in  general  and  on  development  using 
the  Quest  Authoring  System  made  a  significant 
difference  in  the  ability  of  the  team  to  carry  out 
the  project.  The  variety  of  past  experience  tied 
to  this  foundation  of  instruction  made  a  strong 
team.  The  skills  were  there.  The  difficulty  was 
keeping  those  trained  in  place  long  enough  to  do 
the  job.  In  a  few  cases,  members  of  the  team  who 
had  been  drawn  away  for  other  programs  were 
brought  back  to  address  speciGc  issues  and 
work  through  problems  blocking  progress.  The 
schedule  was  set  up  from  the  beginning, 
knowing  that  adjustments  would  need  to  be 
made,  lessons  would  be  learned,  and  problems 
would  arise  that  would  cause  tasks  to  take  longer 
than  expected.  This  built-in  time  allowed  the 
flexibility  needed  throughout  the  course  of  the 
project. 
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USING  HYPERTEXT  IN  COMPUTER  ASSISTED  INSTRUCTION 


Ron  W.  Rhoads,  C.  Q.  Thronesbery 

Hypertext  is  an  emerging  technology  which  allows  the  user  to  create  linkages  among  text, 
graphic,  numeric,  and  other  data  sets.  Systems  are  being  designed  which  will  provide  the 
instructional  designer  with  the  capability  to  create  a  linked  set  of  instructional  materials. 

Once  this  linked  set  of  materials  is  created  by  the  designer  and  developer,  it  can  be  tailored  to 
the  specific  instructional  setting.  The  material  can  be  individualized  for  the  student  by  deleting 
or  creating  links  within  the  existing  materials.  This  tailoring  ability  provides  the  designer  and 
developer  with  a  powerful  tool  to  address  specific  training  problems.  Tailoring  of  instructional 
materials  can  include  linking  to  the  necessary  background  material  for  remediation  when 
necessary.  In  addition  to  this,  all  background  material  (job  and  task  analyses,  media  selection 
tools,  adjunct  instructional  material,  etc.)  can  be  linked  within  a  structured  presentation  format. 

An  appiication  of  the  emerging  technology  of  hypertext  to  computer-assisted  instruction  is 
described.  Specific  system  designs  are  provided  for  a  computer  system  which  illustrates  the 
development  of  the  preservation  system.  With  the  proper  Macintosh  equipment,  a  demonstration 
of  the  system  can  be  provided. 
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Corps  Operations  Decision  Aid,  or  CODA)  which  contains  information  from  Army  Field  Manuals.He 
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System  (AFATDS),  Battalion  Battle  Simulation  (BABAS)  TACFIRE  Training  System  (BTSS),  Corps 
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USING  HYPERTEXT  IN  COMPUTER  ASSISTED  INSTRUCTION 


R.  W.  Rhoads,  C.  G.  Thronesbery 


1.  Introduction 

Hypertext  is  an  emerging  technology  which 
allows  developers  to  create  linkages  among 
text,  graphic,  numeric,  and  other  data  sets. 
Systems  are  being  designed  to  provide 
instructional  designers  with  the  capability  to 
create  multi-linked  packages  of  instructional 
materials.  These  linked  instructional 
packages  can  be  tailored  to  a  specific 
instructional  setting  or  individualized  for  a 
student  by  deleting  and  creating  links  within 
the  existing  materials. 

This  tailoring  ability  provides  the  designers 
and  developers  with  a  powerful  tool  to 
address  specific  training  problems.  Tailoring 
instructional  materials  includes  linking  more 
detailed  background  material  for 
remediation,  linking  organizational  material 
for  the  student  (such  as  hand-outs  or  a 
course  syllabus),  and  linking  evaluation  tools 
like  pre-tests  and  post-tests.  All  background 
material  (e.g.,  job  and  task  analyses,  media 
selection  tools,  adjunct  instructional  material) 
can  be  linked  within  a  structured 
presentation  format. 

Thus,  hypertext  technology  provides  the 
capability  for  easier  and  quicker  access  to 
information,  and  providing  a  more  highly 
structured  informaticn  base.  These  two 
advantages  translate  into  more  efficiem 
design  and  building  of  computer  assisted 
instructional  materials.  This  paper  applies 
the  theme  of  this  year's  conference  to  show 
two  strategies  for  using  hypertext 
technology. 

2.  Overview  of  Hypertext 

Hypertext  is  normal  text  which  has  had  links 
added  to  increase  the  accessibility  of  the 
information.  These  links  are  normally  added 
at  document  construction  time  and  are  part 
of  an  overall  organization  of  the  information 
being  dealt  with.  Typically,  these  links  can 
be  placed  between  text  and  text,  text  and 
graphics,  graphics  and  graphics  -  just  about 
anyplace  where  a  direct  conceptual  tie  is 


necessary  or  desirad.  The  resulting 
tiyprertext  document  is  v.ne  in  which  the 
access  to  information  is  made  quicker  and 
more  efficient. 

Recent  months  have  seen  an  explosion  of 
available  tools  and  packages  for  developing 
hypertext  applications.  These  range  from 
the  highly  graphic  and  multifunctional 
SuperCard  which  runs  on  the  .^.pple 
Macintosh  to  the  mostly  textual  and  simpler 
HyperTies  which  runs  on  a  PC.  For  a  more 
complete  discussion  of  the  tools  available, 
see  Conklin  (1987)  and  Shneiderman  & 
Kearsley  (1989). 

With  all  these  tools,  the  significant  factor  in 
the  design  and  construction  of  instructional 
materials  is  that  they  can  be  used  to  build 
and  change  highly  complex  informational 
structures  quickly.  This  capability  provides 
the  develop>er  with  the  ease  needed  for 
adapting  generic  materials  to  individual 
students. 

A  generic  instructional  systems  design 
process  involves,  in  one  form  or  another,  the 
processes  of  analysis,  design,  development, 
implementation,  evaluation,  and  feedback. 
The  capabilities  of  hypertext  can  be 
exploited  to  increase  the  efficiency  of 
performance  in  each  of  these  phases. 

'  During  the  analysis  phase,  when 
job  and  task  analyses  are 
completed,  a  hypertext  system  can 
be  used  to  link  related  tasks,  training 
objectives,  evaluation  instruments, 
and  adjunct  background  materials. 

During  the  design  phase,  a 
hypertext  system  can  be  used  to  link 
together  those  major  portions  of  the 
training  materials  to  be  developed 
(i.e.,  lesson  plans,  training  manager’s 
guide,  student  hand-outs,  exercises, 
etc.). 


88 


‘  During  the  development  phase,  a 
hypertext  system  can  be  used  to 
track  and  document  tne  system 
being  built.  This  means  that  the 
documentation  for  a  computer 
generated  paper- based  system  can 
easily  be  converted  into  a  CBT 
system  It  is  a  simple  matter  to  use 
the  digital  version  of  documentation 
developed  as  a  source  for  CBT 
presentation.  This  can  be 
accomplished  by  creating  the  links 
necessary  to  use  the  source 
informaiicn  for  the  frame 
presentations. 

During  implementation/ 
evaluation/feedback  phase(s)  a 
hypertext  system  can  be  used  to 
archive  older  versions  of  the  system, 
how  those  versions  have  been 
altered  based  on  feedback,  what 
that  feedback  means,  the  source  of 
the  feedback,  and  the  nature  of  the 
aiierations. 

Although  more  elaborate  instructional  design 
tools  are  possible  (see  Thronesbery  & 
Rhoads  (1991))  this  paper  addresses  the 
application  of  one  specific  aspect  of  the  use 
of  hypertext  systems:  the  use  of  hypertext 
document  bases  as  a  means  for  delivering 
an  on-line  course  syllabus  and  subsequently 
on-line  individualized  remedial  instruction 
packages. 

3.  Hypertext  Instruction 

As  described  by  Thronesbery  &  Rhoads 
(1990a),  Rhoads  &  Thronesbery  (1990),  and 
Thompson  &  Thompson  (1987),  the  use  and 
access  of  information  is  greatly  enhanced 
when  it  is  linked  conceptually.  Large 
informational  data  structures  of  this  type, 
when  used  as  a  set  of  course  reference 
documents,  must  be  presented  to  the 
students  in  a  manner  that  is  understandable 
and  instructional.  This  means  that  the 
material  must  be  ordered  to  match  the 
sequence  of  instruction  in  the  class  or  within 
the  computer  assisted  instructional  units. 
And  it  means  that  the  material  must  be 
selected  (or  excluded)  based  on  its 
relevance  to  the  needs  of  the  class  and  the 


individual  students.  See  Figure  1  *A 
Structured  Hypertext  Document  Set*  for  a 
generic  representation  of  a  set  of  linked 
documentation. 

With  a  fully  developed  document  package, 
the  process  of  individualization  is  a  matter  of 
analyzing  the  specific  learning  style  c*  the 
student,  accommodating  the  content  of  the 
course,  and  evaluating  the  presentation 
modos  of  the  material  (the  basis  of  the 
concept  embodied  by  hypermedia 
presentations).  This  linkage  technology  is 
discussed  in  detail  in  *he  paper  by 
Thronesbery  and  Rhoads  in  this  volume. 
The  addition  or  deletion  of  links  as 
necessary  will  enable  an  instructor  to 
construct  an  individualized  leami.'ig  package 
which  uses  the  specific  course  content,  but 
with  the  proper  links  to  background  or 
remedial  material  that  Tit  the  r.aeds  of  the 
individual  student  Instructional  packages 
structured  in  this  manner  can  bypass  course 
content  the  student  has  already  mastered. 
The  linked  material?  ca':  be  structured  to 
contain  additional  remediation  materials 
which  were  not  a  part  of  the  original  course 
syllabus.  These  structured  hypertext 
instructional  systems  optionally  can  contain 
additional  exercises,  evaluation  instruments, 
and  enrichment  material. 

Instructors  (or  instructional  designers)  will 
also  want  to  use  the  capabilities  of  the 
system  to  construct  a  course  syllabus  by 
extracting  and  linking  materials  for  general 
classroom  use.  By  sek'icting  and  linking 
various  book  chapters,  journal  articles, 
lecture  summaries,  classroom  exercises,  and 
any  other  desired  related  materials,  an 
instructor  can  construct  a  course  syllabus 
which  can  be  made  available  on-line  to  the 
students.  This  on-line  syllabus  can  be  linked 
to  the  material  which  is  generally  available  to 
the  student  as  instructional  materials  as  a 
part  of  the  computer-assisted  instruction.  By 
doing  this,  the  instiuctor  has  provided  the 
student  with  a  set  of  easily  accessible 
materials  which  are  indexed  to  the  CAI 
lessons  and  form  the  bulk  of  the  required 
readinr/research  materials  for  the  class. 
See  Figure  2  'Using  the  Linked  Information*. 
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The  Developer  of  the  system  will  be  making 
decisions  about  the  structure  of  the  material 
to  be  included  in  the  system,  the  speciTtc 
content  of  the  material  to  be  included,  and 
the  level  of  detail  necessary  to  meet  the 
instructional  objectives.  This  means  that  the 
Developer  will  be  asking  questions  like: 

■  'Does  this  reference  text  provide 
a  better  description  of  the  material 
than  any  other?' 

■  'Is  this  particular  article  written  in 
a  manner  that  can  be  understood 
easily  by  students  at  this  level  of 
instruction?' 

■  'Is  this  material  relevant  to  the 
student's  needs  at  this  point  in  the 
instructional  sequence  and  should  it 
be  included?' 

*  'How  does  this  material  relate  to 
the  overall  course  and  the  sequence 
of  presentation  to  the  student  for 
increased  understanding  and 
internalization  of  concepts?' 

•  'Is  this  material  too  complex  or 
too  simple  to  be  of  any  use  to  the 
student  as  primary  information  or  as 
background  information?' 

4.  Tailoring  Hypertext  Instruction 

Consider  the  previously  referenced  diagram. 
Figure  1  'A  Structured  Hypertext  Document 
Sef,  which  depicts  the  manner  in  which 
materials  are  linked  to  provide  a  structured 
hypertext  environment  for  the  instructor, 
class,  and  student. 

This  material  is  linked,  concept  to  concept. 
This  highly  complex  structure  provides  the 
users  of  the  instructional  system  with  a  basic 
set  of  materials  for  immediate  access.  This 
material  can  be  accessed  in  a  read-only 
mode,  or  materials  can  be  cut  from  here  to 
be  pasted  elsewhere,  or  material  can  be 
added  as  necessary  and  linked  to  the 
existing  data  What  is  not  provided  in  this 
configuration  is  an  entry  point  into  the 
material.  This  is  the  tool  which  the  instructor 
must  derive  based  on  the  needs  of  the  class 


and  individuai  students.  From  this 
perspective,  a  class  syllabus  and  an 
individualized  remedial  study  plan  are 
isomorphic.  They  both  provide  linked  sets  of 
entry  points  into  the  complex  and  highly 
structured  hypertext  document  base.  See 
Figure  3  The  Syllabus  Provides  Entry  Points 
to  Structured  Hypertexf  and  Figure  4 
'Constructed  Remediation  Links  for  Each 
Studenf. 

These  two  diagrams  show  how  the  same 
document  set  can  be  used  for  two  different 
purposes  by  taking  advantage  of  the  linked 
nature  of  the  material.  The  syllabus  is  used 
to  provide  the  student  with  a  view  of  the 
entire  course  and  the  material  related  to  it 
Students  receive  an  immediate  indication  (rf 
the  material  the  instructor  considers 
significant  and  an  indication  of  the  amount 
and  complexity  of  the  material  used  in  the 
course. 

As  shown,  constructed  remediational 
materials  may  be  highly  elaborate  and  draw 
on  all  the  course  materials,  or  they  may  be 
quite  simple  when  necessary  and  illustrate  a 
single,  narrowly  focused  idea  In  either 
case,  the  ease  with  which  the  instructor  can 
construct  the  package  makes  it  much  more 
feasible  to  develop  a  number  of  remediation 
packages  at  periodic  intervals  for  each 
student.  It  is  conceivable  that  intelligent 
processing  could  supplarrt  the  instn>ctor  in 
this  process  by  analyzing  student  responses 
and  tracking  document  access  time  by 
student  to  determine  whether  specific 
documents  have  been  used  in  the 
study/performance  process. 

Students  who  access  this  material  will  need 
only  travel  the  desired  navigational  links 
provided  by  the  instructor.  Instructors  will 
find  it  a  simple  matter  to  build  on-line, 
accessible  remediational  structures  for 
individual  students  by  the  simple  addition  of 
linked  data  sets  which  reintroduce  difficult 
material,  reinforce  previously  discussed 
material,  and  assess  the  effectiveness  of  the 
remediation  when  it  is  complete.  Students 
will  not  need  to  have  the  linkages  made 
explicit,  as  long  as  they  are  aware  of  the 
nature  of  the  material  being  delivered.  This 
contextual  information  will,  of  course,  be 
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significant  in  helping  the  student  develop  an 
understanding  of  the  material  which  is 
available  discussing  this  topic  and  the 
manner  in  which  this  material  is  related. 

5.  Conclusions 

Hypertext  is  a  young  technology 
(Thronesbery  &  Rhoads  1990b). 
Nonetheless,  it  has  potential  applicability  to 
many  of  the  mundane  information  access 
problems  which  make  training  programs 
difficult  to  deal  with  and  irritating  to  use. 
Unlike  traditional  database  management 
programs  which  normally  require  the 
development  of  a  prespecified  data 
construct,  hypertext  systems  are  able  to  use 
data  available  in  many  different  formats. 
This  means  that  the  instructional  designer 
has  the  freedom  to  link  many  different  kinds 
of  information  storage  and  presentation 
modes.  A  chapter  in  a  book  can  be  linked 
to  a  picture  which  can  be  linked  to  a  table 
which  can  be  linked  to  a  journal  article  which 
can  be  linked  to  a  graph.  The  hypertext 
developer  does  not  worry  about  the  form  of 
the  data,  only  the  content. 

This  freedom  also  provides  the  instructional 
designer  with  the  ability  to  develop  useable 
classroom  tools  like  the  ones  described  in 
this  paper:  the  syllabus  builder  and  the 
remedial  exercise  construct.  Hypertext 
document  bases  can  be  used  to  link  large 
and  varied  amounts  of  information.  Once 
built,  that  information  can  be  tapped  as  a 
resource  for  students  using  on-line 
computer-assisted  instruction.  The  tapping 
mechanism  can  be  designed  to  accomodate 
group  use  (as  in  a  class  syllabus)  or 
individual  use  (as  in  remedial  exercises 
constructed  for  a  single  student). 
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Textbook  C  -  Ch.  1, 6, 8 
Textbook  D  -  Ch.l,  2, 7 
Textbook  E  -  Ch.  4, 5 
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MANAGING  CONTRACTOR  DEVELOPMENT  OF  INTERACTIVE 

COURSEWARE  (ICW) 

T.  Kent  Thomas 

As  interactive  courseware  (ICW)  becomes  more  widely  recognized  as  a  viable  training  media  that  can 
provide  cost-effective  training,  more  and  more  development  is  being  done  via  contract.  However,  the 
process  used  to  develop  ICW  is  significantly  different  and  more  complex  than  that  used  to  develop 
training  for  more  traditional  media  such  as  platform  instruction,  written  text,  or  audiovisual  modules.  As 
Gloria  Gery  states  in  her  landmark  book.  Making  CBT  Happen,  'As  a  training  or  project  manager,  you 
should  know  process  is  the  key  critical  success  factor.  Manage  it  that  way.  ” 

The  author  will  present  the  ICW  development  process  defmed  in  the  draft  Military  Handbook, 

Manager's  Guide  for  Development,  Acquisition,  and  Management  of  Interactive  Courseware  for 
Military  Training.  He  will  relate  how  deviations  from  that  process  can  and  have  caused  significant 
problems.  The  documentation  developed  during  this  process  forms  the  major  contract  milestones  and  is 
the  foundation  of  government  quality  assurance  reviews. 

The  author  will  present  lessons  learned  in  five  year's  experience  in  managing  six  ICW  contracts  for 
development  of  over  50  courses.  Successful  contracting  for  ICW  must  begin  with  a  clear  definition  of 
the  process  to  be  followed  and  the  deliverables  (both  the  documentation  and  courseware).  Further, 
precisely  estimating  the  scope  of  an  ICW  development  effort  is  virtually  impossible  due  to  the  number  of 
'ariables  involved.  But,  there  are  strategies  that  allow  procurement  of  effective  training  within  budget 
via  firm,  fixed  priced  contracts. 


MR.  THOMAS  was  formerly  an  Air  Force  Education  and  Training  officer  and  a  computer  systems 
requirements  analyst.  With  a  B.A.  from  Carson-Newman  Culieg?,  he  is  now  completing  a  Masters  in 
Instructional  Technology  from  Utah  State  University.  From  1981-1985,  he  was  a  Training  Systems 
Analyst  at  HQ  ATC/TTX.  He  served  as  ATC  focal  point  for  student  management  functions  in  the 
Advanced  Personnel  Data  System  (APDS)  and  project  officer  for  the  Branch  Level  Training  Management 
System  (BLTMS),  ATC's  largest  operational  computer-managed  instruction  (CMI)  system.  During 
1985-86,  he  served  as  Chief,  Training  Initiatives  Branch  at  HQ  TAC/LGQT,  where  he  helped  plan 
TAC's  ICW  initiative.  He  activated  the  4400th  Maintenance  Training  Flight  (MTF),  TAC's  maintenance 
ICW  development  and  management  center,  assuming  command  in  1986.  The  Tactical  Air  Forces  (TAC, 
PACAF,  USAFE,  AFRes,  and  ANG)  now  have  over  30  interactive  videodisc  (IVD)  maintenance  courses 
implemented  throughout  the  world,  operating  on  approximately  300  training  systems.  Over  30  more 
courses  are  in  work,  by  both  contractors  and  4400  MTF  personnel.  The  4400  MTF  has  developed  nine 
IVD  courses  internally,  including  the  winners  of  the  Nebraska  Videodisc  Design/Production  Group's 
award  for  best  govemment/military  achievement  in  both  1990  and  1991.  He  is  now  the  Director  of 
Commercial  Courseware  Development  at  Allen  Communications,  Inc.,  Salt  Lake  City,  Utah. 
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MANAGING  CONTRACTOR 
DEVELOPMENT  OF 
INTERACTIVE  COURSEWARE 
(ICW) 

T.  Kent  Thomas 

Introduction.  Contracting  for  the 
development  of  training  materials  can  be  a 
frustrating  experience  with  disappointing 
results.  These  conditions  are  compounded 
when  the  complexity  of  interactive  courseware 
(ICW)  is  added.  Contracting  for  ICW 
development  has  traditionally  been  a  risky 
venture,  for  both  the  contractor  and  the 
Government.  There  are  all  too  many  examples 
of  cost  overruns,  defaults  or  judgments,  and 
marginal  products.  Perhaps  it  is  because  there 
are  so  many  diverse  theories  of  instructional 
design  that  get  translated  into  very  subjective 
media  treatments  and  presented  to  students  by 
fairly  complex  computer  software.  Or,  perhaps 
it  is  because  no  one  truly  knows  how  people 
learn.  In  any  event,  designing  and  developing 
effective  ICW  is  often  more  of  an  art  form  than 
a  scientific  discipline.  Just  imagine  trying  to 
contract  for  the  original  design  of  a  work  of  art 
-  especially  with  a  fully  competitive,  firm  fixed- 
price  contract!  I  sure  wouldn't  want  to 
arbitrarily  award  the  contract  to  the  lowest 
bidder. . . . 

About  five  years  ago  1  was  contacted  by  an 
associate  to  see  if  I'd  participate  in  a  working 
group  to  develop  a  "generic"  contracting 
package  for  ICW.  I  quickly  concurred,  since  I 
was  then  involved  with  one  ICW  contract  that 
clearly  had  some  shortcomings  and  was  defining 
another.  I  definitely  could  use  the  help  and 
advice,  even  if  I  could  contribute  nothing  but 
"lessons  learned".  Consequently,  I  had  the 
privilege  of  being  a  "charter  member"  of  the 
Joint  Service  Action  Group  (JSAG)  to  Develop 
Interactive  Courseware  (ICW)  Data  Item 
Descriptions  (DIDs).  DIDs  are  the  contracting 
documents  to  define  the  format  and  content  of 
any  data  (information,  regardless  of  form  or 
media)  delivered  via  contract.  This  ad  hoc 
group  of  ICW  developers  and  end-users  from  all 
the  services  proved  to  be  an  excellent  forum  for 
sharing  problem  areas,  prospective  solutions, 
and  lessons  learned.  Meeting  quarterly,  it  has 


been  a  very  productive  organization,  producing 
almost  20  standard  ICW  DIDs,  a  glossary  of 
ICW  terms,  the  revisions  necessary  to 
incorporate  ICW  in  MIL-STD-1379,  Military 
Training  Programs,  and  most  recently  a  draff 
(as  of  the  date  of  this  paper)  Military 
Handbook,  Manager's  Guide  for  Development , 
Acquisition  and  Management  of  Interactive 
Courseware  for  Military  Training. 

Perhaps  the  most  interesting  aspect  of 
participating  in  this  JSAG  was  in  discovering 
the  commonality  -  everyone  was  encountering 
similar  problems.  We  soon  found  that  everyone 
meeting  success  was  also  following  a  similar 
design  and  development  process,  with  similar 
design  documents  being  delivered  and  reviewed. 
Corrective  actions  that  solved  a  problem  for  one 
participant  were  often  tried  elsewhere,  and 
quickly  became  common  practice.  This  theme 
of  practical,  "real-world"  validation  was 
prevalent.  Even  the  DIDs  that  we  developed 
were  "test  driven"  as  "one-time  DIDs"  on  ICW 
contracts  before  formal  submission  and  approval 
for  DoD-wide  use.  There  is  now  a  common, 
proven  approach  for  defining  and  managing 
contractor  development  of  ICW.  I  would 
NEVER  state  that  this  is  the  only  ICW 
development  process  that  works.  But,  I  fully 
agree  with  Gloria  Gery  (1988)  that  the  process 
used  to  develop  ICW  is  the  key  critical  success 
factor. When  it  is  impossible  to  completely 
define  the  end  product,  as  it  is  with  ICW,  you 
must  define  the  process  that  will  be  used  to 
produce,  test,  and  evaluate  it.  So,  the  keys  to 
success  in  contracting  for  ICW  are  in  defining 
the  end  products  required  as  clearly  and 
concisely  as  possible,  plus  the  process  the 
contractor  will  follow  to  develop  those 
products.  The  development  process  then 
becomes  the  foundation  of  the  Government's 
quality  assurance  (QA)  program  to  ensure  the 
products  meet  the  originally  defined 
requirement.  It  is  in  the  QA  of  deliverables  that 
the  Government  is  most  actively  involved. 

This  paper  will  briefly  describe  the  ICW  design 
and  development  process  contained  in  the 
Military  Handbook  previously  mentioned.  Note 
that  the  handbook  also  contains  guidance  on 
front  end  analysis  for  ICW  and  maintenance  of 
the  ICW  after  development.Those  topics  will 
not  be  addressed.  Similarly,  this  paper  will 
address  only  key  points,  based  on  practical 
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experience,  and  is  not  an  in-depth  treatment  of 
the  subject.  Refer  to  the  referenced  documents 
for  more  details.  (I  strongly  recommend  the 
Military  Handbook  to  anyone  involved  in  ICW 
development  or  management.)  The  focus  of  this 
paper  is  on  key  points,  lessons  learned,  and 
specific  tips  of  practical  value. 

Statement  of  Work.  A  clear,  concise 
statement  of  work  (SOW)  is  the  foundation  of 
the  entire  ICW  development  project  and  all 
management  controls  (both  contractor  and 
Government)  since  it  establishes  and  defines  all 
contractor  efforts.  For  general  guidelines  on 
SOWs,  see  MIL-HDBK-245,  Preparation  of 
Statement  of  Work,  and  MIL-HDBK-248, 
Acquisition  Streamlining.  A  Government 
sponsored  survey  of  the  ICW  industry  identified 
several  factors  that  affect  the  contractor's  ability 
to  price  an  ICW  design  and  development 
contract  accurately  (i.e.  these  factors  increase 
the  technical,  cost,  and  schedule  risks  for  both 
the  contractor  and  the  Government).  Each 
factor  could  have,  and  should  have,  been 
addressed  in  the  statement  of  work.  Let's 
examine  them  in  the  order  of  their  importance 
and  potential  impact. 

Course  Scope  and  Objectives. 

Precisely  estimating  the  scope  of  an  ICW 
development  effort  is  virtually  impossible  due 
to  the  number  of  variables  involved.  The 
length,  complexity,  subject  matter  and  target 
audience  impact  the  scope,  just  to  name  a  few  of 
the  factors  involved.This  difficulty  in  defining 
the  scope  of  ICW  development  efforts  can 
increase  the  contractor's  risk  to  the  point  that  it 
is  very  difficult  to  use  firm,  fixed  price 
contracts.  This  leaves  the  Government 
vulnerable  to  any  unforeseen  problems  and  the 
resultant  prices  increases.  However,  you  can 
shift  the  risk  associated  with  vague  scope  away 
from  the  contractor  and  to  the  Government, 
significantly  decreasing  the  chance  of  problems. 
Lets  look  at  one  method  of  estimating  the 
course  length  and  complexity  for  a  technical 
skill  such  as  aircraft  maintenance. 

First,  define  the  training  objectives  as  clearly  as 
possible,  including  enabling  objectives.  Then, 
estimate  the  time  required  to  teach  each 
objective  to  a  single  'average'  student  using 
one  or  more  of  the  following  traditional 


methods,  including  all  tests  or 
evaluations;  classroom  lecture  and  task 
demonstration;  guided  and  then  independent 
practice;  hands-on  performance  testing  with 
actual  equipment.  Use  a  panel  of  experienced 
instructors  (minimum  of  three)  familiar  with  the 
subject  matter  and  target  population  to  make 
these  estimates,  if  possible.  Try  for  a  consensus 
on  the  estimated  time.  If  you  can't  get  a 
consensus  then  get  them  to  establish  a  range  of 
high,  low  and  average  times.  The  smaller  the 
range  the  better. 

Next,  have  the  panel  estimate  the  percentage  of 
the  total  training  time  that  each  method  is  used 
for  each  objective.  For  example,  consider  a 
maintenance  task  that  takes  an  estimated  two 
hours  to  teach.  Classroom  lecture  describing 
the  task,  its  importance,  and  demonstrating  how 
to  perform  it  takes  approximately  30%  of  the 
time.  Student  practice  in  performing  the  task, 
with  initially  full  assistance  then  decreasing 
levels,  takes  50%  of  the  time. Evaluating  the 
student's  performance,  hands-on,  will  take  the 
remaining  20%  of  the  time.  This  example 
yields  36  minutes  of  lecture  and  demonstration, 
60  minutes  of  practice  with  decreasing  levels  of 
assistance,  and  24  minutes  of  hands-on  testing. 
Repeat  this  process  for  every  training  objective. 
Then  add  up  the  time  each  instructional  method 
would  be  used  in  the  entire  course  (i.e.  all  the 
objectives). 

This  process  describes  the  content,  course 
length,  and  the  complexity  of  the  course  in 
traditional  terms.  Next,  this  must  be 
'translated'  to  ICW.  The  JSAG  defined  three 
levels  of  ICW  presentations  that  describe  both 
its  complexity  and  its  interactivity,  as  follows: 

LEVEL  1  -  Baseline  presentation.  This  is  the 
lowest  level  of  interactive  courseware 
development.  It  is  basically  a  knowledge  or 
familiarization  lesson,  in  linear  format  (one  idea 
after  another),  used  mainly  for  introducing  an 
idea  or  concept.  The  trainee  has  little  control  of 
what  is  seen  (minimum  trainee  interactivity). 

The  two  types  of  baseline  presentation  are:  1) 
Video  and  minor  text  presentation  2)  Graphics 
and  minor  text  presentation. 

LEVEL  2  -  Medium  Simulation  Presentation. 
This  medium  presentation  level  involves  the 
recall  of  more  information  than  a  baseline  Level 
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1  presentation  and  allows  the  trainee  to  have 
increased  control  over  the  lesson  presentation 
(e.g.,  use  of  touch  screen  or  lightpen  to  rotate 
switch).  A  moderate  degree  of  simulation  is 
used  in  the  presentation.This  presentation 
provides  the  following: 

1)  Combined  information  and  skills 

lessons 

2)  Moderate  degree  of  programming 

3)  Trainee  interactivity  with  various 
I/O  devices 

4) CMI  to  track  and  analyze  student 
performance 

5)  Normally  combines  video  and 
graphics  presentations. 

LEVEL  3  -  High  Simulation  Presentation.  The 
highest  level  entails  aspects  of  both  Level  1  and 
Level  2  while  using  the  fullest  abilities  of 
interactive  courseware.  Every  possible  subtask 
is  analyzed  and  presented  for  full,  on-screen 
interaction,  similar  to  that  used  in  aircraft 
simulator  technology.  The  trainee  can 
determine  areas  in  which  further  training  is 
desired.  This  presentation  provides  the 
following: 

1)  Primarily  used  for  procedural 
tasks/skilJs 

2)  High  student  interactivity 

3)  Extensive  branching  capability  (falls 
short  of  artificial  intelligence) 

4)  Maximum  remediation  opportunity 
(e.g.,  multiple  responses  measure  degree  of 
error  and  give  relevant  responses) 

5)  Real  time  event  simulation  with 
minor  equipment  limitations  (e.g.,  timing 
sequences  of  start-up,  switch  changes) 

6)  Capability  to  interface  with  other 
output  devices 

7)  Exhaustive  CMI  capability. 

These  presentation  levels  correlate  quite  well  to 
the  traditional  instructional  methods  used  for 
estimating,  above.  Subjects  taught  by 
classroom  lecture  and  demonstration  can  be 
presented  by  Level  1  ICW.  Guided  and 
independent  practice  methods  can  be  presented 
by  Level  2.  Subjects  evaluated  by  hands-on 
methods  can  be  presented  by  Level  3.  But, 
research  strongly  indicates  that  ICW  achieves 
equivalent  levels  of  student  proficiency  in 
significantly  less  time  (Fletcher,  1990; 

DeBloois,  1984;  ATCP  50-4).  I'd  recommend 
that  you  be  somewhat  conservative,  until  you've 


validated  the  efficiency  of  ICW  in  your  training 
environment.  So,  initially  reduce  your 
estimates  for  each  presentation  level  by  no  more 
than  20  percent,  when  converting  estimated 
traditional  training  time  to  estimated  ICW 
length. 

Defme  the  scope  in  the  SOW  by  describing  the 
target  population,  listing  the  training  objectives, 
and  defining  the  hours  of  ICW  (based  on  the 
average  student  contact  time  at  small  group 
tryouts)  to  be  developed  for  each  presentation 
level.  Include  the  JSAG  definitions  of 
presentation  levels,  since  the  ICW  industry  is 
becoming  familiar  with  them.  Include 
incremental  options  in  the  request  for  proposals 
to  procure  additional  training  that  can  be 
executed  on  or  after  award  to  take  care  any 
inaccuracies  or  underestimations. 

I  make  no  claims  on  the  initial  accuracy  of  the 
results  provided  by  this  method,  also  due  to  the 
number  of  variables  involved.  The  accuracy  of 
the  panel's  estimates  is  the  prime  one,  and 
appears  dependent  on  the  number  of  instructors 
used  to  develop  the  estimates  and  their 
experience  in  teaching  the  same  or  similar 
subject  matter.  This  approach  merely  adds 
structure  to  a  subjective  evaluation  and 
judgment  -  i.e.  it's  a  "soft  process".  Continue 
to  refme  and  apply  it  and  the  accuracy  of  your 
estimates  should  improve.  In  our  experience  it 
works  just  as  well  as  some  much  more  elaborate 
estimating  models  and  it  is  much  easier  to  use 
and  explain. 

IVedisposition  to  Media  Features  and 
Instructional  Approaches  This  second  factor 
contributing  to  risk  is  essentially  a  strong 
preference  on  your  part  that  should  be  clearly 
communicated  to  the  contractor.  Your 
preconceptions  will  directly  influence  how  you 
view  the  contractor's  work.  The  solution  to  this 
problem  is  to  either  require  compliance  with  an 
ICW  style  guide  that  defines  your  preferences 
or  have  one  developed  that  establishes  the 
conventions  or  standards  to  be  used.  The  style 
guide  should,  as  a  minimum,  address  these 
areas:  sample  course  and  lesson  architecture, 
video  and  graphics  screen  composition,  student 
interaction  with  the  courseware,  screen  color 
schemes,  menus,  icons,  testing  strategies,  and 
language  conventions.  Leave  flexibility  in 
instructional  design,  and  focus  on  "ICW 
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operation,  look  and  feel".  But,  do  not 
compromise  on  basic  principles  of  instructional 
design,  media,  learning  theory,  cognitive 
psychology,  etc. 

On  our  first  ICW  contract,  we  didn't  have  a 
style  guide  in  the  contract  and  didn't  know  we 
needed  one  -  until  the  first  courseware  was 
under  development!  We  had  accepted  the 
responsibility  for  quality  assurance,  acceptance 
testing,  and  configuration  management 
(including  updates  after  final  delivery)  of  a 
previously  awarded  contract.  The  contract  was 
the  outgrowth  of  an  independent  research  and 
development  project  as  a  technology 
demonstrator  to  assess  the  effectiveness  of 
interactive  videodisc  (IVD).  What  we  had 
accepted  management  responsibility  for  -  more 
than  a  definitive  product  -  was  a  process  of 
producing  ICW  based  on  software  and 
instructional  design  principles  and  practices. 
Basically,  we  could  not  "require"  the  contractor 
to  do  more  than  make  sure  the  ICW  complied 
with  the  course  outline,  flow  diagrams,  scripts, 
and  storyboards  that  we  approved  at  each  stage 
during  development.  Even  the  requirements  for 
these  design  documents  were  vague  (the  JSAG 
DIDs  weren't  available  then). 

The  initial  courses  were  a  rehosting  of  the 
technology  demonstrations  (cockpit  procedure 
simulations  and  the  corresponding  motion  video 
demonstrations).  With  the  contractor's  full 
cooperation,  we  mutually  defined  how  and  then 
converted  them  to  stand-alone  exportable 
training  packages.  The  quality  of  the 
courseware  gradually  improved  as  the 
contractor  progressed  to  new  courses  and 
improvements  continued  over  the  term  of  the 
contract.  This  was  partially  due  to  the 
contractor's  selective  and  voluntary  cc.npliance 
with  a  style  guide  that  we  developed  as  we  went 
along.  The  contractor  built  temf  ates,  used 
them,  and  incorporated  recommendations  to 
make  additions  or  enhancements  to  each  course 
as  timelines  and  budget  allowed. 

We  accepted  similar  responsibilities  on  a  second 
contract  shortly  after  the  first  one.  This  was 
our  first  opportunity  to  provide  input  on  an 
ICW  SOW,  so  we  requiied  the  joint 
development  and  approval  of  a  style  guide  after 
contract  award.  We  worked  closely  with  the 
contractor  to  jointly  develop  a  style  guide  based 


on  research  into  the  ICW  literature,  experience 
with  other  media,  other  available  style  guides, 
and  limited  hands-on  experience  from  our  in- 
house  development  efforts.  There  were  several 
problems  with  this  approach,  even  though  the 
conventions  were  mutually  agreed  upon. 

First,  the  style  guide  lacked  detail  in  several 
important  areas  so  it  poorly  defined  our 
expectations.  Second,  the  wording  was  not 
restrictive  enough  and  was  still  open  to  too 
much  interpretation.  Third,  it  did  not 
differentiate  between  essential  items  and  those 
that  are  "nice  to  have".  Finally,  it  was 
developed  without  our  seeing  prototype  lessons 
based  upon  it.  When  we  finally  saw  some  of 
the  interpretations  in  the  first  courseware,  we 
didn't  like  them  too  well,  but  it  was  too  late  to 
change  it  on  this  firm,  fixed  price  contract.  We 
learned  from  our  mistakes  -  and  revised  the 
style  guide. 

This  revision  more  clearly  defined  our 
preferences,  having  seen  things  we  didn't  like. 

It  also  incorporated  an  ICW  evaluation  checklist 
to  ensure  reviewers  focused  their  attention  on 
what  we  considered  the  critical  items.  We 
provided  it  as  government  furnished 
information  on  the  next  contract.  It 
communicated  better  but  it  still  wasn't 
definitive  enough  -  we  had  not  clearly  identified 
the  mandatory  items  to  the  contractor.  Further, 
we  only  stated  in  the  SOW  that  the  contractor 
would  "develop  ICW  in  accordance  with  the 
style  guide."  We  couldn't  enforce  it,  though 
there  was  more  compliance  than  with  the 
previous  contract.  At  least  we  were 
communicating  and  the  development  process 
was  working. 

The  fourth  ICW  contract  (and  third  version  of 
the  style  guide)  worked  better.  We  defined  the 
guidance  in  the  style  guide  as  standards  and 
conventions,  with  a  "sliding  scale"  of 
restrictiveness.  First,  some  items  are  absolutely 
mandatory  standards  and  cannot  be  waived. 
Second,  some  items  are  considered  standards 
but  can  be  waived.  Third,  we  defined  items  as 
conventions  that  will  be  followed  unless  there  is 
a  technical  or  instructional  reason  for 
noncompliance.  The  remaining  items  were 
considered  informational  and  compliance  was 
optional.  We  also  changed  the  ICW  evaluation 
checklist  to  reflect  these  levels  of  restriction  for 
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items  that  were  not  optional.  Our  style  guide 
had  evolved  into  a  document  describing  a 
common  student  interface,  instructional  and 
media  design  principles,  a  sample  course 
structure,  and  the  student  data-gathering 
requirements.  There  was  still  considerable 
latitude  for  creativity  in  design  -  even  the 
sample  course  structure  was  optional.  Then  we 
stated  in  the  SOW  that  "the  contractor- 
developed  ICW  shall  reflect  the  design 
requirements  of  the  attached  style  guide". 

We  required  the  contractor  to  perform  an 
internal  quality  control  review  of  each 
deliverable  or  document  using  the  criteria 
identified  in  ICW  evaluation  checldist  from  the 
style  guide.  We  also  stated  that  "the  contractor 
shall  comply  with  the  guidelines  specified  in  the 
style  guide,  or  request  a  waiver  in  writing 
through  the  contracting  officer  to  the  Air  Force 
user  organization,  including  supporting 
rationale  and  any  recommendations  for 
improving  or  changing  the  requirements  of  the 
style  guide. "  We  also  provided  sample  courses 
as  Government  furnished  information  to 
illustrate  the  style  guid  ■  requirements.  It's 
communicating  pretty  well  -  we  even  used  this 
version  of  the  style  guide  on  another  contract 
without  revising  it! 

We  strongly  recommend  that  you  provide  a 
style  guide  as  an  attachment  to  the  SOW  and 
require  the  contractor  to  comply  with  it  or 
request  written  waivers.  If  you  do  not  have  a 
style  guide,  then  request  copies  from  others  - 
since  they've  been  developed  by  several 
different  organizations.  I  have  never  been 
refused  when  I  requested  a  copy  of  a  style  guide 
from  another  military  organization.  We  provide 
ours  freely  and  they  respond  in  kind.  Either  use 
the  best  one  you  can  find,  create  your  own 
using  the  best  from  each  one,  or  pay  the 
contractor  to  develop  one  prior  to  actually 
beginning  courseware  development.  If 
possible,  provide  ICW  samples  that  illustrate 
your  preferences.  If  you  cannot,  then  require 
the  contractor  to  develop  a  prototype  lesson  that 
incorporates  these  guidelines,  also  prior  to 
actual  courseware  development.  This  will  show 
their  interpretation  of  those  guidelines  and  you 
can  then  revise  them  as  necessary,  as  long  as  the 
contractor  agrees.  In  our  experience,  style 
guides  are  essential.  If  you  do  not  provide  one, 
the  contractor  must  either  use  their  own  or 


develop  one.The  contractors  prefer  that  you  tell 
them  what  you  want  "up-front",  since  they 
don't  enjoy  surprises  either!  Meanwhile,  our 
style  guide  is  under  revision  again, 
incorporating  more  lessons  learned  about  ICW 
instructional  design,  development  and  contract 
management.... 

Availability  and  Stability  of  Government 
Systems,  Equipment,  and  Personnel.  This  is 
one  risk -contributing  factor  that  you  may  have 
little  control  of,  in  some  cases.  It  involves  two 
closely  related  aspects,  including  reasonable 
contractor  access  to  Government  bases,  systems, 
equipment,  technical  data,  and  subject  matter 
experts  (SMEs).But,  no  less  important  is  the 
stability  of  those  resources.  Is  the  configuration 
and  operational  capability  of  the  systems  and 
equipment  changing  frequently?  Will  the  same 
SMEs  or  other  key  personnel  be  available 
throughout  the  entire  project?  Where? 

The  stability  of  weapons  systems,  support 
equipment  and  technical  data  has  been  a 
constant  problem  for  us.  Since  a  large  part  of 
our  ICW  is  to  support  "differences  training"  and 
conversion  to  new  weapons  systems,  they  are 
constantly  changing.  Our  first  contract  didn't 
address  the  issue  at  all.  However,  since  it  was 
the  prime  contractor  that  was  also  developing 
the  courseware,  they  often  knew  of  forthcoming 
changes  before  we  did,  and  changed  their 
development  plan  accordingly.  There  were  still 
some  "surprises"  but  the  contractor  voluntarily 
incorporated  the  changes  into  the  courseware  - 
but  not  the  documentation  in  all  cases. 

The  second  contract,  also  with  a  prime 
contractor,  still  did  not  adequately  address  these 
issues  (we  hadn't  been  "bitten"  on  the  first 
contract  at  that  point).  It  merely  required  the 
ICW  to  agree  with  the  previously  reviewed  and 
approved  design  documents.  However,  we 
were  able  to  get  tech  data  changes  incorporated 
in  the  design  documents  during  the  development 
process  in  some  cases.  For  example,  a  tech  data 
change  after  the  flow  diagrams  were  approved 
was  incorporated  in  the  scripts  and  storyboards 
-  (and  consequently  in  the  ICW)  but  the  flow 
diagrams  were  never  completely  updated.  They 
were  annotated  during  red-line  reviews,  but  new 
ones  were  never  produced  incorporating  the 
edits. 
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The  largest  problem  on  this  contract  was  that 
the  aircratit  was  late  coming  off  the  assembly 
Ine,  as  was  some  support  equipment  and  all  the 
tech  data.  Luckily,  since  it  was  the  prime 
contractor  the  Government  was  not  liable  for 
damages  or  cost  overruns  for  failure  to  provide 
aircraft  and  tech  data  to  the  contractor  at  the 
scheduled  times.  We  couldn't  readily  provide 
them  with  something  yet  to  be  delivered  to  us. 

We  were  required  to  provide  access  once  the 
aircraft  were  delivered,  and  we  did.  On  the  last 
delivery  increment,  we  received  large  tech  data 
changes  after  small  group  tryouts  and  prior  to 
final  acceptance.  The  contractor  did  not 
incorporate  the  changes.  As  it  was,  the 
contractor  probably  lost  money  on  this  ICW 
contract  due  to  schedule  slippage.  Impact  to  the 
government,  in  addition  to  the  ICW  being 
delivered  late,  was  that  the  last  increment 
needed  updates  as  soon  as  it  was  delivered.  The 
Government  awarded  the  ICW  contract  to  the 
prime  contractor  in  order  to  make  the  ICW 
contractor  share  some  of  the  risk.  In  this  case 
both  parties  paid!  Such  is  the  nature  of  ICW 
development  for  a  brand  new  weapons  system! 

But,  no  other  maintenance  training  devices  were 
available  that  simulated  '■ircraft.  The  o.nly 
aj  irnau  -'c  *0  '  was  to  dedicate  more  actual 

aircraft  ‘  use  as  tramiug  devices. 

The  third  contract,  also  to  a  prime  contractor, 
required  the  contractor  to  develop  an 
incremental  delivery  schedule  based  on  the 
availability  of  validated  tecnnical  data  and 
efficient  utilization  of  his  resources.  It  also 
required  the  courseware  to  be  technically 
correct  at  small  group  tryouts.  Up  to  that  point 
the  contractor  had  to  incorporate  ail  technical 
data  changes.  The  Government  was  vulnerable 
to  tech  data  changes  only  after  tryouts.  But,  we 
didn't  require  the  contractor  to  go  back  and 
update  the  documentation,  mstead,  they  had  to 
correct  all  errors  that  we  identified  during 
reviews  and  submit  a  corrected  copy  within  30 
days.  So,  even  though  the  ICW  has  been 
current  upon  delivery  (to  this  point  we  have  not 
had  a  tech  data  change  during  the  interim 
between  tryouts  and  final  delivery),  the  design 
documents  have  been  current  only  in  some 
cases. 

The  fourth  and  subsequent  contracts  require  that 
both  the  ICW  and  the  design  documentation  be 
current  at  small  group  tryouts.  This  may  appear 
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to  place  undue  risk  on  the  contractors. 

However,  we  use  prime  contractors  for 
emerging  weapons  systems,  where  both  the 
delivery  schedule  and  course  content  are  the 
most  dynamic.  We  use  con^titive  contracts 
for  older,  less  dynamic  weapons  systems.  We 
also  require  the  contractor  to  perform  a  brief 
front-end  analysis  to  refine  and  revalidate  the 
requirements,  and  to  make  any 
recommendations  for  changes  in  course  content. 
Then,  the  contractor  would  propose  an 
incremental  delivery  schedule  based  upon  the 
availability  of  validated  technical  data  and 
efficient  utilization  of  his  resources.  In  a 
nutshell,  we  pay  the  contractor  to  examine  the 
potential  for  changes  and  to  develop  a  schedule 
that  will  try  to  avoid  the  problem  areas.  We  are 
also  supportive  of  changing  the  sequence  that 
specific  courses  will  be  delivered,  to  coincide 
with  availability  and  stability  of  all  required 
resources  (including  tech  data). 

We  insist  that  the  ICW  be  technically  current  at 
small  group  tryouts.  We've  found  that  you  are 
essentially  wasting  your  time  in  performing 
small  group  tryouts  on  an  'out  of  date*  course 
for  several  reasons.  First,  the  students  are  very 
sensitive  to  "face  validity"  -  -  if  the  ICW  isn't 
accurate  and  current,  they  "tune  out" 
immediately,  invalidating  the  results.  Second, 
it  is  more  difficult  to  objectively  evaluate  the 
ICW  against  a  previous,  "frozen"  baseline  that 
no  longer  exists  in  the  "real  world"  than  if  the 
ICW  is  current.  For  example,  if  there  a  need  to 
resolve  a  question  about  how  a  piece  of 
equipment  actually  operates,  you  and  the 
contractor  can  mutually  observe  it  in  operation. 
How  could  this  be  done  when  the  operation  of 
the  equipment  had  changed  but  the  design  of  the 
ICW  had  not?  Further,  if  any  substantial 
changes  are  required  to  make  the  ICW  current  - 
which  it  must  be  prior  to  implementation  -  then 
it  may  need  to  go  to  small  group  tryouts  again. 
Why  do  it  twice  unless  absolutely  necessary? 
Finally,  it  encourages  the  contractor  to  develop 
ICW  that  is  easily  updated,  should  changes 
occur.  Some  ICW  design  techniques  can 
significantly  impact  the  ease  of  updates.  For 
example,  video  still-frame  equipment 
simulations  can  easily  be  updated  with 
authoring  changes,  if  the  needed  video  exists. 
This  encourages  the  contractor  to  develop  visual 
databases  in  case  of  changes.  These  databases 
greatly  reduce  ICW  configuration  management 


and  update  problems  during  development  and 
after  acceptance  by  the  Government. 

Subject  matter  experts  (SMEs)  are  also  a  critical 
part  of  the  ICW  design  and  development 
process,  both  in  "quality  and  quantity. "  They 
should  have  the  highest  and  most  current 
technical  qualifications  available,  work  well 
with  others,  and  have  good  communications 
skills.  If  at  all  possible,  they  should  have 
enough  projected  tenure  to  complete  the  project. 
A  change  in  SMEs  during  the  design  and 
development  can  have  a  significant  disruptive 
effect  on  the  effort,  as  the  new  SME  tries  to 
learn  everything  that  took  place  before.  They 
will  likely  want  to  make  changes,  so  expect  it. 
Do  not  allow  them  to  change  the  design  unless 
both  you  and  the  contractor  agree.  Should  they 
find  technical  inaccuracies,  those  should  be 
corrected,  of  course. 

Sufficient  contractor  access  to  SMEs  is  also 
critical  factor,  though  the  degree  may  vary 
based  up  n  the  course  content.  Be  sure  to 
clearly  specify  the  SME  support  that  you  can 
provide  in  the  statement  of  work,  and  be 
prepare'*  to  discuss  it  with  the  contractor(s).  I 
guarantee  that  it  is  a  sensitive  issue  if  the 
contractor  is  developing  completely  new 
training  materials.  It  has  been  a  "bone  of 
contention"  at  some  time  on  every  ICW  contract 
I've  ever  been  involved  with.  There  are  no 
specific  guidelmes  that  I  can  give  on  "how 
much  is  enough."  In  general,  give  as  much 
SME  support  as  you  can  afford,  but  monitor 
how  the  contractor  utilizes  the  SMEs  and  make 
sure  that  the  time  is  productive. 

Note  that  if  any  Government  reviewer  changes, 
not  just  the  SMEs,  it  can  have  a  disruptive 
effect  on  the  contract.  This  reinforces  the  need 
for  specified  guidelines,  such  as  a  style  guide. 
Guidelines  greatly  reduce  the  subjectivity  in 
product  evaluation  for  everyone  concerned. 

Review  and  Approval  Processes.  The  review 
and  approval  process  should  be  defined  in  the 
SOW  and  based  on  the  deliverables  from  an 
iterative  ICW  design  process.  The  ICW  should 
be  designed  in  stages  of  increasingly  greater 
detail,  with  review  and  approval  by  the 
Government  prior  to  the  contractor  proceeding 
to  the  next  stage.  Similarly,  the  delivery  of  the 
final  ICW  should  be  in  increments  (units. 


modules,  courses,  etc),  if  at  all  possible.  This 
allows  "lessons  learned"  by  both  the  contractor 
and  the  Government  to  be  incorporated  in  the 
next  incremental  delivery.  Do  not  assume  that 
it  is  either  more  efficient  or  effective  to  "review 
it  all  at  one  time. "  One  ICW  contract  I'm 
aware  of  did  not  require  either  of  these 
processes.  The  contractor  was  allowed  to 
proceed  from  approved  lesson  design  outlines  to 
delivery  of  all  the  ICW  at  one  small  group 
tryout  without  a  formal  Government  review  of 
any  further  design  documents.  The  Government 
representatives  were  not  aware  of  any  serious 
problems  until  the  contractor  failed  to  meet  the 
ICW  delivery  date  (for  a  multitude  of  problems 
including  availability  of  SMEs  and  equipment). 
The  contractor  eventually  made  it  to  small 
group  tryouts  over  a  year  late,  only  for  the 
Government  to  cancel  the  tryouts  the  first 
morning  because  the  ICW  was  totally 
imacceptable.  As  of  the  date  of  this  paper,  the 
ICW  still  has  not  been  accepted,  though  it  made 
it  through  tryouts.  It  is  still  being  revised  -  as  it 
approaches  being  two  years  behind  schedule  - 
and  the  Government  has  acknowledged  it  will 
likely  be  of  marginal  quality. 

The  ICW  design  and  development  process 
defined  in  the  cited  Military  Handbook  makes 
use  of  the  JSAG's  original  efforts  in  defining 
standard  ICW  DiDs.  However,  the  JSAG  was 
later  tasked  by  the  Office  of  the  Assistant 
Secretary  of  Defense,  Force  Management  and 
Persoimel  (OASD/FM&P)  to  incorporate  the 
ICW  and  the  DIDs  in  MIL-STD-1379,  Military 
Training  Programs,  while  it  was  being  revised. 
This  incorporation  revised  the  DIDs  and  merged 
them  with  other  new  training  development 
DIDs.  where  appropriate.  The  Military 
Handbook  includes  only  these  new  DIDs, 
though  the  intent  and  purpose  are  the  same  as 
the  originals.  Appendix  B  of  the  Military 
Handbook  describes  how  to  invoke,  tailor  and 
use  these  revised  DIDs.  The  new  DIDs  are 
listed  in  parentheses  after  the  appropriate 
descriptions  below.  The  titles  of  the  original 
DIDs  are  used  in  this  discussion  since  they  are 
more  descriptive  of  the  content  and  purpose. 

The  DlDs  that  we  have  used  consistently  are 
listed  below  in  the  general  sequence  that  they 
should  be  developed,  delivered  and  reviewed. 
Note  that  there  are  other  DIDs  available.  Your 
contracting  office  or  data  item  manager  can 
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provide  further  assistance  in  selecting  and 
tailoring  DlDs.  Also  note  that  you  should 
thoroughly  review  any  of  these  DIDs  to  delete 
any  requirement  for  information  that  you  do  not 
need.  They  contain  several  requirements  that 
are  unique  to  the  particular  military  services  and 
any  unnecessary  information  will  increase  the 
cost  for  no  purpose. 

ICW  Contract  Work  Plan,  DI-MGMT-80549, 
provides  a  detailed  description  of  management 
tasks  which  must  be  completed  to  fulfill 
guidelines  for  development  of  ICW.  It 
describes  the  management  structure,  milestones, 
lessons  to  be  developed,  travel  schedules, 
personnel  and  organization,  validation 
process/criteria,  and  quality  control  activities. 
(DI-ILSS-81070,  Training  Program 
Development  and  Management  Plan) 

ICW  Design  Strategy,  DI-ILSS-80547, 
describes  the  general  design,  structure,  content, 
instructional  strategy,  and  media  treatment  of 
the  ICW.  (DI-ILSS-81091,  Instructional  Media 
Design  Report) 

ICW  Flow  Diagrams,  DI-ILS$-80548,  provide 
a  detailed  map  of  the  intended  logic  of  the 
subject  courseware.  They  include  all  defined 
lesson  tasks,  information  frame  or  sequence, 
decision  points,  branching  options,  and 
remediation.  (DI-ILSS-81091,  Instructional 
Media  Design  Report) 

ICW  Video  Shot  Support  Plan,  DI-ILSS-80802, 
describes  all  resources  (personnel,  equipment, 
facilities,  etc.)  required  for  support  plus  all 
needed  information  to  plan  and  coordinate  ICW 
video  production,  to  include  dates  and 
durations.  (DI-ILSS-81091,  Instructional 
Media  Design  Report) 

ICW  Script-Storyboards,  DI-ILSS-80546, 
provide  a  blueprint  for  the  production  of  the 
lew.  Includes  scripting  information  and  visual 
representations  of  the  materials,  plus  all 
required  directions  for  the  video  director, 
programmer  and  instructional  designer.  (Dl- 
ILSS-81092,  Instructional  Media  Package) 

ICW  Video  Shot  List,  DI-ILSS-80803,  describes 
all  video  shots  required  to  produce  the  ICW, 
sorted  in  a  sequence  to  minimize  camera 


setups.  (DI-ILSS-8 1092,  Instructional  Media 
Package) 

ICW  Edit  Decision  List,  DI-ILSS-80804, 
describes  the  planned  video  editing  by 
specifying  start  and  end  points  of  resource  video 
and  audio  in  the  sequence  they  will  be 
assembled.  (DI-ILSS-81092,  Instructional 
Media  Package) 

ICW  Terminology  Document,  DI-ILSS-80550, 
lists  all  abbreviations,  acronyms,  nmemonics, 
etc.  used  in  naming  or  describing  modules, 
lessons,  data  files,  subroutines,  screen  displays 
and  variables  in  the  ICW.  (DI-ILSS-8 1092, 
Instructional  Media  Package) 

lew  Manager's  Guide.  DI-ILSS-80552,  briefly 
describes  the  content  and  structure  of  the  ICW, 
how  to  install  and  operate  it,  and  how  to 
produce  student  management  reports.  (DI- 
ILSS-8 1096,  Training  System  Utilization 
Handbook) 

The  design  strategy,  flow  diagrams  and  script- 
storyboards  are  the  most  important  and  most 
used  documents,  since  they  define  the  actual 
course  content.  Require  that  they  are  developed 
for  each  incremental  delivery  of  ICW  and 
delivered  in  sequence.  Each  one  must  be 
reviewed  and  approved  before  the  contractor 
proceeds  to  developing  the  next  one.  Do  not 
formally  accept  the  documents  when  initially 
submitted,  merely  approve  them  and  give  the 
contractor  authority  to  proceed.  Note  that  i  f 
you  formally  accept  the  document  then  you  have 
just  baselined  the  design  of  the  course  and  the 
contractor  cannot  be  required  to  make  any 
significant  change  in  the  course  design.  These 
documents  should  be  thoroughly  reviewed  by 
both  instructional  designers  and  subject  matter 
experts  when  initially  received,  generating 
discrepancies  that  must  be  corrected  and  any 
recommendations  for  improvement.  If  the 
course  will  simulate  a  piece  of  equipment, 
process,  or  procedure,  I  strongly  recommend 
that  you  (and  the  contractor,  if  possible)  "walk 
through"  the  flow  diagrams  "live"  with  the 
applicable  equipment,  process  or  procedure  to 
verify  the  accuracy  of  the  simulation  logic 
involved  prior  to  approving  the  document.  If 
the  simulatiou  is  either  very  detailed  or  very 
complicated,  consider  "walking  through"  the 
storyboards  also. 


These  dcKuments  should  remain  in  draft  until 
after  small  group  tryouts  of  the  courseware,  and 
available  for  review  at  every  interim  review 
point.  This  allows  the  iterative  instructional 
design  process  to  further  refine  the  instructional 
strategy  as  necessary  while  making  sure  there 
are  no  significant  changes  in  course  content. 

Do  not  require  the  flow  diagrams  to  exactly 
match  the  initial  design  strategy,  for  example. 
Allow  the  contractor  to  make  changes  in  the 
instructional  design  as  necessary  as  the  next 
stage  is  developed  -  as  long  as  you  review  and 
approve  the  change  and  all  preceding  documents 
are  updated.  If  you  "force"  the  contractor  to 
follow  the  previously  approved  design 
documents  verbatim,  you  limit  their  creativity 
and  the  quality  of  the  fmal  product.  On  the 
other  hand,  do  not  allow  them  to  make  changes 
just  "because  they  wanted  to. "  Require  that  they 
explain  how  the  change  in  the  instructional 
design  will  improve  the  ICW.  This  will  also 
help  you  conceptualize  and  understand  their 
"view"  of  the  ICW.  Do  not  allow  them  to  make 
a  significant  reduction  in  the  scope  of  the 
delivery  increment,  as  defined  in  the  SOW  by 
objectives,  estimated  length,  and  presentation 
levels.  That  is,  unless  you  agree  that  it  is 
appropriate  and  the  contractor  offers  "tradeoffs" 
such  as  increasing  the  number  of  optional 
practice  exercises  in  the  ICW  or  increasing  the 
next  increment.  Make  sure  you  document  any 
reductions  in  scope  on  a  given  increment,  and 
that  the  contracting  officer  is  aware  of  and 
agrees  with  the  change  (or  has  delegated 
authority  to  you). 

The  video  shot  support  plan,  video  shot  list, 
and  edit  decision  list  are  used  only  when 
required,  such  as  for  interactive  videodisc. 

They  should  also  be  developed  for  each  delivery 
increment.  They  should  be  submitted  and 
revised  only  as  necessary  for  acceptance.  They 
should  not  remain  in  draft,  since  they  are  "one¬ 
time"  requirements.  The  shot  support  plan  is 
usually  produced  after  the  flow  diagrams,  while 
the  script-storyboards  are  being  developed,  to 
tell  you  what  they  plan  for  the  production.  This 
allows  you  the  lead  time  to  coordinate  all 
support  and  have  it  ready  when  needed.  The 
shot  list  is  to  ensure  the  video  production  is 
both  complete  and  efficient.  It  should  be 
comprehensive  (i.e.  listing  every  required  shot), 
sorted  to  minimize  camera  setups,  and  cross- 


referenced  to  the  script-storyboards.  It  should 
be  reviewed  and  accepted  prior  to  the  first  day 
of  scheduled  video  production.  The  edit 
decision  list  is  to  ensure  the  on-line  edit  session 
is  well-planned  and  efficient,  and  to  allow  you 
the  opportunity  to  review  the  video  footage  that 
will  be  assembled  (if  you  desire),  prior  to  it 
being  edited.  It  should  be  reviewed  and 
accepted  prior  to  the  first  day  of  scheduled 
editing. 

The  next  stage  is  authoring,  where  the  actual 
ICW  is  assembled,  incorporating  all  graphics 
screens  and  video,  to  include  all  branching 
logic.  The  actual  courseware  should  be 
reviewed  at  least  twice.  The  first  review  (alpha 
test  or  individual  tryouts)  should  be  done  to 
ensure  the  ICW  agrees  with  the  approved  script- 
storyboards  and  flow  diagrams,  it  operates 
correctly,  it  is  complete  and  ready  for  small 
group  tryouts.  You  should  also  have  it 
reviewed  by  a  SME  (preferably  one  involved 
with  previous  reviews)  to  verify  technical 
accuracy.  You  should  have  two  other 
documents  delivered  in  draft  at  this  point,  as 
required.  The  terminology  document  completes 
the  course  design  documents,  and  is  very 
valuable  when  updates  are  required.  It  should 
be  developed  for  each  delivery  increment.  The 
manager's  guide  is  the  overall  documentation  on 
how  to  use  and  manage  the  KTW.  If  all  the 
ICW  will  be  incorporated  into  one  course, 
require  the  manager's  guide  to  be  initially 
submitted  with  the  first  incremental  delivery, 
and  revised  with  each  subsequent  one.  If  the 
delivery  increments  are  "stand-alone*  modules 
or  courses,  require  a  complete  manager's  guide 
for  each.  If  there  are  significant  numbers  of 
discrepancies  at  this  preliminary  ICW  review, 
you  should  consider  another  review  to  ensure 
their  correction  prior  to  small  group  tryouts. 

The  second  review  (beta  test  or  small-group 
tryouts)  is  similar  to  that  for  any  other  training 
package  or  media.  Use  your  normal  criteria  for 
selecting  the  target  audience  and  validating  the 
results.  However,  the  contractor  should  be 
present  to  observe  the  ICW  in  operation  so  he 
can  understand  any  problems  that  may  appear. 
As  mentioned  previously,  validation  criteria 
should  be  stated  in  either  the  SOW  or  the 
contract  work  plan.  If  the  ICW  validates  in 
accordance  with  the  stated  criteria,  then 
acceptance  is  a  matter  of  the  contractor 
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correcting  any  discrepancies  documented  at 
small  group  tryouts,  and  your  verification  of 
their  correction.  However,  you  should  place  a 
clause  in  the  statement  of  work  that  "the 
Government  reserves  the  right  to  repeat  all  steps 
of  the  ICW  design  and  development  process 
until  assured  that  the  ICW  meets  the  specified 
and  approved  technical  and  instructional 
requirements."  This  "closes  the  loop"  on  the 
formative  evaluation  process  and  requires  the 
contractor  to  make  any  changes  necessary  until 
the  ICW  validates  successfully. 

Factors  Outside  the  Statement  of  Work. 

There  are  two  other  factors  that  impact  the 
overall  success  of  the  ICW  development  effort. 
They  are  not  a  part  of  the  actual  statement  of 
work,  but  should  be  incorporated  in  the 
contract.  The  first  is  the  type  of  contract  and 
the  second  is  the  payment  schedule. 

Due  to  the  difficulty  in  specifically  defining 
ICW  contracts  in  the  past,  there  were  many 
types  of  contracts  used.  Each  has  its  advantages 
and  disadvantages.  The  ICW  Military 
Handbook  has  an  excellent  discussion  of  the 
alternatives.  In  general,  the  most  desirable  type 
of  ICW  contract  and  the  type  with  the  least  risk 
to  the  government  is  a  firm,  fixed-price 
contract.  This  type  of  contract  pays  the 
contractor  a  fixed  price  for  the  development  of 
the  product,  regardless  of  the  effort  expended. 
Their  efficiency  is  "rewarded"  by  increased 
profits,  while  they  bear  full  risk  for 
inefficiencies.  It  has  been  difficult  in  the  past  to 
award  firm,  fixed-price  contracts  because  it  was 
difficult  for  both  the  Government  and  the 
contractor  to  estimate  the  scope  of  the  effort  and 
the  corresponding  price.  However,  the  use  of 
this  common  ICW  development  process, 
standardized  DIDs  and  definition  of  ICW 
presentation  levels  has  changed  that. 

There  are  definite  trends  in  the  number  of 
contractor  labor  hours  being  bid  for  ICW  design 
and  development.  In  our  experience,  and 
collaborated  by  other  JSAG  members,  you  can 
expect  to  see  the  following  ranges  of 
development  hours  per  student  contact  hour 
(average  at  small  group  tryouts)  for  interactive 
videodisc  ICW :  Level  1 ,  200-250  labor 
hours/ICW  hour;  Level  2,  400-450;  Level  3, 
600-650.  Though  we  have  no  direct 
experience,  the  reported  ratios  for  ICW  using 


computer  graphics  and  no  audio  are:  Level  1, 
100-150;  Level  2,  250-300:  Level  3,  400-450. 
Note  that  these  development  ratios  are 
decreasing  as  the  industry  gains  more 
experience  in  ICW  development  and  contracting 
for  it.  Technology  is  always  producing  more 
refined  tools  for  ICW  development  also.  Note 
also  that  the  labor  for  Level  3  ICW  is  directly 
related  to  the  complexity  of  the  equipment  being 
simulated  and  the  amount  of  "free  play" 
provided.  For  example,  a  "total  free  play" 
cockpit  simulation  of  a  new,  complex  fighter 
aircraft  could  exceed  2000  hours  of  labor  per 
contact  hour.  Be  sure  to  also  define  equipment 
being  simulated  and  the  amount  of  free  play 
required  as  accurately  as  possible  in  the  SOW, 
if  dealing  with  complex  equipment. 

The  second  factor  not  in  the  statemet  of  work  is 
the  payment  schedule.  You  do  not  want  to  pay 
too  much  money  too  early  in  the  ICW  design 
and  development  process  to  "protect  yourself* 
in  case  of  contractor  problems  or  default.  On 
the  other  hand,  if  the  contractor  must  obtain 
loans  to  meet  their  payroll,  these  costs  will  be 
passed  on  to  the  government.  A  "partial 
payment"  schedule,  in  which  the  contractor  is 
paid  a  specified  percentage  at  specific 
milestones  works  well  to  keep  the  contractor 
"motivated  to  perform."  These  milestones  must 
be  based  on  delivery  and  Government 
acceptance  or  approval  of  products  -  do  NOT 
simply  pay  the  contractor  at  specified  periods  of 
time.  You  must  also  balance  the  "partial 
payments"  against  the  contractor's  labor 
expenditures  when  deciding  on  a  payment 
schedule. 

We've  had  very  good  experiences  with  a  partial 
payment  schedule  for  each  delivery  increment, 
and  with  each  delivery  increment  being 
separately  priced.  For  example,  a  contract  for 
five  courses  would  pay  roughly  one-fifth  the 
total  contract  price  for  each  course.  Upon 
Government  approval  of  the  design  strategy  for 
a  given  course,  the  contractor  would  receive 
25  %  of  the  total  for  that  course.  Upon 
government  approval  of  the  script-storyboards 
the  contractor  receives  the  next  25%,  with  the 
remainder  (50%)  paid  upon  final  Government 
acceptance  of  the  course  and  all  the  supporting 
documentation.  This  approach  pays  the 
contractor  frequently  and  steadily  -  as  long  as 
he  delivers  acceptable  products.lt  also  spreads 
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Conclusion.  While  managing  the  development 
of  ICW  is  always  a  challenge,  it  can  be  done 
successfully.  The  keys  to  success  are  a  clear 
and  concise  definition  of  what  you  want  and  a 
systematic  process  of  development,  review  and 
approval.  These  factors  should  be  addressed  in 
the  statement  of  work  for  the  contract.  The 
type  of  contract  and  the  payment  schedule  can 
also  reduce  the  rist  to  the  govertunent. 


both  the  technical  and  fmancial  risk  across  the 
duration  of  the  contract.  At  any  given  time 
both  the  amount  of  money  and  product  at  risk  is 
relatively  small,  should  a  major  problem  occur. 
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video-based  Training  Analysis  and  Rapid  Prototyping  of  CBT 

William  F.  Jorgensen,  William  R.  Terrell,  Cecil  L.  Wakelin 

Current  training  demands  have  greatly  increased  the  requirement  for 
interactive  technologies  for  development  and  presentation  of 
courseware.  However,  data  requirements  for  interactive  training 
technologies  are  not  supported  by  traditional  Instructional  Systems 
Development  (ISD)  techniques.  Traditional  ISD  methodologies  for 
identification  of  training  requirements  typically  produce  linear  text 
descriptions  of  job,  tasks,  and  task  elements  and  do  not  describe  the 
complex  interactions  of  total  performance  within  the  environmental 
context  of  a  job.  Interactive  training  technologies  have  advanced 
training  capability  to  a  level  that  extremely  complex  training  events 
and  environments  are  possible.  The  linear  text-based  approach  results 
in  continuous  re-visits  to  the  job  to  collect  data  for  IVD  or  other 
interactive/pictorial  media,  adding  to  the  cost  and  resource 
requirements  of  the  development  process.  Instruct onal  design 
methodologies  must  advance  at  a  pace  commensurate  with  instructional 
technology.  This  advance  must  include  the  structuring  of  data  bases 
capable  of  handling  extremely  complex  task  performance  and 
environmental  interactions.  These  data  bases  should  include  audio  and 
pictorial,  as  well  as  linear  text  oriented  data.  An  analysis 
technology  is  needed  which  is  capable  of  prescribing  detailed 
requirements  for  the  interactive  pictorial-based  courseware. 

This  paper  will  describe  the  development  of  a  video/audio  data  base 
structured  to  handle  complex  task  performance  and  environmental 
information.  It  will  also  describe  an  analysis  process  that  is  capable 
of  quickly  prototyping  audio  video-based  courseware  and  converting  data 
into  specifications  for  the  development  of  learning  objectives, 
instructional  events,  media,  media  formats,  and  specifying  training 
systems  features. 
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William  F.  Jorgensen,  William  R.  Terrell,  Cecil  L.  Wakelin 


State-of-the-art  technology  has 
had  tremendous  impact  on  the  job 
environment.  Pilots,  operators, 
and  other  performers  are  faced 
with  more  threats,  more 
information,  more  capabilities, 
and  the  need  to  make  decisions 
at  a  much  more  rapid  pace  than 
ever  before.  Training 
individuals  to  cope  with  this 
much  more  complex  job 
environment  is  pushing 
instructional  technology  to  keep 
pace.  Computer  based  training 
(CBT)  has  moved  toward  increased 
student  interactivity  with 
complex  instructional 
environments.  The  technology 
that  has  so  drastically  changed 
the  work  and  training 
environments  also  provides  che 
technological  capability  so 
instructional  design  can  keep 
pace.  (See  Figure  1>  Computer- 
based  technology  and  training 
analysts  have  been  intermingling 
for  years,  and  use  of  computers 
to  analyze  or  develop  is  not  new 
to  instructional  designers  and 
training  analysts.  There  have 
been  many  attempts  and  some 
successes  using  computers  to 
analyze  job/task  data.  (See 
Figure  2)  A  partial  list 
includes  TASCS,  CASDAT,  CASAT, 
ETRAN,  ETASC,  NOTADS,  CATIDS, 
and  more.  The  latest,  and  most 
promising  traditional  ISD 
automated  process,  is  the  Joint 
Service  ISD/LSAR  Decision 
Support  System,  (See  Figure  3) 
None  of  these  systems,  however, 
directly  support  the  development 
of  state-of-the-art  interactive 
instruction.  Each  system  is 
based  on  analysis  methodologies 
which  produce  linear  text 
descriptions  of  job/task  data 
appropriate  to  text-based 
courseware,  media,  and  training 
materials . 


Requirements  for  video/audio 
data 

Traditional  instructional  design 
methodologies  favor  the  use  of 
text  based  presentation  of  each 
individual  element  of  the  job 
environment.  When  pictorially 
based  courseware  is  required  for 
training,  and  a  linear  text- 
based  data  analysis  has  been 
done,  a  re-analysis  for  more 
data  is  required  to  provide  the 
courseware  screens,  pictures, 
and  descriptions  of  the  job 
interactivity.  If  we  take  a 
look  at  a  typical  CBT  design 
process,  such  as  this  one  for 
IVD  (see  Figure  4)  ,  we  notice 
numerous  revision  loops.  These 
loops  represent  several 
activities  in  an  interactive 
design  and  development  process. 
These  loops  also  represent  a 
backwards  chaining  process  to 
the  data  collection  procedure, 
representing  costly  re-analysis 
activity  to  collect  more 
pictorial-based  data  from  the 
job/task  environment. 

Interactive  instructional 
technology  is  based  on  the 
fundamental  principle  that  the 
student  should  interact  with  the 
job  environment  which  is 
represented  or  simulated  in  the 
instructional  environment.  Job 
and  task  data  collection  must 
capture  the  total  job 
environment  including  the 
interactions  among  the  elements, 
the  core  of  which  is  visual  with 
verbal  support  information. 
Imagine,  if  you  will,  complex 
job  environments  such  as  an 
airplane  cockpit  (Figure  5)  or  a 
combat  information  center. 
Operators  in  these  environments 
have  multiple  displays,  multiple 
response  choices,  and  are 
orchestrating  a  host  of 
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concurrent  events.  Add  a 
variety  of  intelligent  and 
environmental  external  threats, 
plus  the  requirement  to  interact 
with  other  team  members,  and  you 
have  an  idea  of  the  complexity 
of  the  job  environment. 
Traditional  instructional  design 
methodologies  would  examine  each 
job  and  each  task  required  to 
perform  that  job.  The  training 
analysis  focuses  on  training  the 
student  to  perform  each  discrete 
task,  but  leaves  to  the  trainer 
the  job  of  teaching  the 
composite  or  concurrency  of  the 
tasks.  To  paraphrase  a  point 
Dr.  M.  David  Merrill  made  in  a 
presentation  at  the  last  TITE 
conference,  (see  Figure  6)  there 
is  something  wrong  when  we 
propose  to  use  technology  to 
enhance  and  improve  instruction, 
but  advocate  up  to  several 
hundred  times  more  effort  to 
develop  the  new  technology  based 
instruction.  His  point,  of 
course,  is  to  use  the  technology 
to  improve  analysis  and  reduce 
design  and  production  time,  as 
well  as  improve  the 
instructional  process.  Dr. 
Merrill  is,  of  course, 
attempting  to  do  something  about 
the  courseware  development 
process  with  his  ID  Expert  2.0. 
However,  Merrill  and  Li  state 
that  "most  of  the  content 
information  required  by  a 
transaction  is  available 
directly  from  the  content 
structure  elaborated  during 
content  analysis  process." 
Current  technology  also  provides 
the  tools  for  total  content  and 
instructional  designers  to 
collect  data  on  the  complexity 
of  the  job  environment  and  then 
analyze  that  data  to  define 
those  features  of  an 
instructional  environment 
required  to  interactively  train 
the  job. 

State-of-the-art  instructional 
designers  are  conducting 


research  and  developing 
interactive  audio  video 
technology  which  can  collect  and 
organize  job  and  task  data, 
analyze  the  data,  and  specify 
the  training  system  required. 
The  focus  of  this  paper  is  such 
a  process  which  collects  and 
analyzes  complex  job  and  task 
data.  (Figure  7)  The  specific 
methodology  described  involves 
the  use  of  video  recording 
technology  to  create  a 
video/audio  data  base.  The  use 
of  automated  coding  procedures 
to  describe  basic  cue, 
cognition,  and  response  activity 
makes  it  possible  for  decision 
algorithms  to  analyze  the  data. 


Using  video  data 
Several  companies  and 
organizations  have  been  using 
video  as  a  data  collection 
process  since  the  inception  of 
the  portable  video  camera.  Most 
of  this  work  has  been  focused  on 
workload  or  human  factors 
analysis,  and  most  of  it  has 
been  collected  in  a  laboratory 
or  simulator  setting.  One  such 
example,  is  the  data  collection 
at  the  SIMNET  D  facility.  Ft. 
Knox,  Kentucky.  The  purpose  of 
the  SIMNET  video  cameras  is  to 
observe  and  evaluate  the 
activity  of  students  using  the 
SIMNET  M-1  tank  simulators  from 
a  remote  observer  station.  The 
video  tracks  are  observed  in 
real  time,  and  can  be  reviewed 
later  to  provide  feedback  to 
students.  Dynamics  Research 
Corporation  (DRC)  has  been  using 
the  video  data  collected  during 
simulator  operation  to  examine 
operator  workload. 

Workload  analysis  procedures 
select  a  video  data  stream  from 
the  observer's  video  recordings, 
identify  start  and  stop  events 
in  a  task,  replay  the  video,  and 
observe  and  record  the  operator 
activity.  (See  Figure  8)  The 


recorded  information  is  analyzed 
for  workload  determination.  DRC 
is  using  the  recorded  video 
streams  of  data  to  examine  the 
workload  of  the  operator  under 
various  conditions,  such  as  when 
the  operators  are  using 
experimental  or  prototype 
equipment .  An  example  is  when 
new  control  panels  are  installed 
which  change  the  tank  driver' s 
or  commander's  procedures . The 
video  recording  concept  is  the 
baseline  for  video  based 
job/task  analysis. 

Developing  the  video  audio  data 
base 

DRC  is  currently  building  a 
prototype  data  collection  and 
analysis  procedure  for  use  by 
the  Naval  Training  Systems 
Center  (NTSC)  for  the 
specification  of  training  system 
features.  Development  of  this 
procedure  consists  of  creating  a 
data  base  which  includes  video 
and  sound  tracks  (see. Figure  9), 
and  the  creation  of  codes  to 
label  the  activities  recorded  on 
the  videotape  and  sound  tracks. 
Algorithms  are  developed  which 
analyze  the  codes.  A  reporting 
format  specifies  the  required 
training  device  features.  The 
data  coding  step  is  comprised  of 
two  basic  elements:  (see  Figure 
10)  a  pre-analysis  to  collect 
job  data  structure  and  data 
collection  and  coding. 
Development  of  a  job  data 
structure  establishes  the 
following  (see  Figure  11) : 

•What  is  the  system? 

•What  are  the  appropriate 
subsystems,  components,  and 
other  system  definers? 
•What  is  the  function 
performed  by  the  person- 
machine  interface  to  be 
analyzed? 

•What  is  part  of  and  what 
is  external  to  the 
operational  and  the  person- 


machine  environment? 

For  example,  the  system  may  be  a 
fighter  aircraft,  the  subsystem 
may  be  an  air  search  radar,  and 
the  components  may  include  a 
multi-function  display  control 
panel  and  a  cursor  control  on 
the  aircraft  control  stick  (see 
Figure  12) .  The  environment 
description  may  include  the 
number  of  people  in  a  cockpit,  a 
description  of  cues  to  be 
expected  both  internal  and 
external  to  the  cockpit,  the 
communication  requirements,  and 
the  operational  parameters  or 
interfaces  of  the  radar  to  other 
subsystems . 

Development  of  a  job  data 
structure  establishes  the 
guidelines  and  boundaries  for 
the  analysis.  For  example, 
specifying  which  cues  are 
influenced  by  the  pilot,  and 
which  are  not.  This  becomes 
important  in  the  data  collection 
and  analysis  process  described 
next,  because  cues  which  are 
external  to  the  operational 
environment  of  the  pilot-machine 
interface  are  used  to  define 
segments  of  behavior. 

Data  to  be  collected  (see  Figure 
13a)  includes  a  video  recording 
of  behavior,  including  close-ups 
of  controls  and  indicators.  If 
audio  is  part  of  the  job,  this 
will  be  collected  as  well.  This 
may  mean  two  or  more  video/audio 
tracks.  To  collect  this  data, 
microcameras  linked  via  fiber 
optics  to  recorders,  will  record 
the  pre-determined  data.  Audio 
is  recorded  and  synchronized 
with  the  video  data. 

Once  the  data  is  collected,  the 
operator  who  performed  the  task 
adds  a  crucial  aspect  to  the 
data  base,  the  cognitive,  e.g., 
decision-making,  data.  (See 
Figure  13b)  To  gather  this 
information,  the  recorded 


audiovisual  job/task  information 
is  replayed,  using  a  dedicated 
audio  track,  the  operator 
narrates  his  or  her  reasons  for 
doing  things,  what  they  were 
thinking  while  doing  them,  the 
sequence  of  the  events  and 
reason  for  it,  what  they  are 
doing  concurrently,  and  a  host 
of  other  pre-determined  data. 
This  recording  process  uses  a 
start/stop  control  during  the 
audio  recording,  which  allows 
the  operator  to  stop  the 
video/audio  data  stream  if  the 
cognitive  description  stretches 
beyond  the  physical  time  limit 
or  length  of  the  recorded  video 
data.  The  cognitive  track  is 
produced  by  interaction  of  the 
operator  with  a  trained  analyst, 
who  is  cognizant  of  the  needed 
information  for  each  segment  of 
the  audiovisual  data  stream. 
Data  also  includes  any  still 
pictures,  graphics,  or  text 
which  will  support  the 
video/audio  data  base  (see 
Figure  13c) . 

Use  of  the  vldeo/audio  data  base 
A  set  of  cue,  cognition,  and 
response  codes  were  developed 
from  review  of  research 
performed  in  the  fields  of 
psychology,  human  factors,  and 
educational  technology.  Codes 
were  selected  which  represented 
specific  types  of  behavior  and 
conditions  in  each  domain  (see 
Figure  14) .  For  example,  a 
visual  cue  on  a  radar  screen  may 
be  Vg  for  videographic,  B3  would 
represent  Bloom' s  third  level  of 
cognition,  and  S3-PBM  would 
represent  Simpson' s  taxonomic 
rating  of  psychomotor  activity 
at  level  3  and  involve  the  use 
of  a  push  button  multi-function 
display.  The  codes  are  assigned 
by  an  analyst  when  playing  back 
the  video  and  sound  tracks  using 
real  time,  slow  motion,  and  stop 
action  as  appropriate.  The 
codes  are  keyed  to  the  video  for 
retrieval,  review,  or  other 


manipulations.  To  analyze  the 
codes,  algorithms  have  to  be 
developed  which  compare, 
contrast,  weight,  match,  or  in 
some  way  make  decisions  relative 
to  the  code  content  and  the 
combined  meanings,  depending  on 
the  products  requigred.  The  data 
codes  can  be  rapidly  scanned  by 
computer.  Rules  of  analysis  or 
algorithms  are  used  by  the 
computer  to  analyze  the 
alphanumeric  codes,  rather  than 
job  data,  to  select  training 
system  features,  sequence 
training  requirements,  group 
content,  prepare  behavioral 
statements  or  to  meet  many  other 
requirements . 

The  goal  of  the  NTSC  training 
features  selection  project  is  to 
specify  training  device  features 
based  on  job/task  behaviors  and 
cognitive  content.  The 
hypothesis  was  that  too  many 
things  are  happening 
concurrently  in  an  aircraft 
cockpit  to  make  rational 
decisions  based  on  analysis  of 
individual  linear  text-based 
tasks.  What  we  expect  to  find 
is  a  heavy  concurrency  of 
activity,  both  psychomotor  and 
cognitive,  and  when  this 
concurrency  is  considered  in  the 
analysis  process,  these  results 
vary  greatly  from  the 
traditional  approach.  For 
example,  a  night  attack  pilot, 
in  addition  to  operation  of  the 
aircraft  in  low  level 
navigation,  must  contend  with 
weather  conditions,  other 
external  environmental  factors, 
night  vision  goggles,  FLIR, 
RADAR,  target  acquisition, 
target  identification,  weapon 
selection,  arming  and 
deployment,  hostile  threats,  and 
an  occasional  emergency 
procedure .  To  develop  an 
accurate  portrait  of  the 
behaviors,  environment,  and 
conditions,  the  analyst  must 
perceive  all  of  the  elements  in 
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their  concurrent  relationships 
and  translate  them  to  some 
intelligible  description  of  the 
required  training  device 
features.  Figure  15  illustrates 
the  training  system  features 
analysis  process.  Note  that  the 
video/audio  codes  are  analyzed 
directly  by  the  process  and  are 
converted  into  training  device 
featiires.  Iteration  is  provided 
for  by  the  retrieval  and  review 
key  feature  built  into  the 
coding  process. 

The  video/audio  data  base 
process  has  implications  for  the 
future  of  courseware 
development.  (See  Figure  16) 
DRC  has  already  developed  for 
NTSC  a  computer  aided  training 
analysis  tool  called  Learning 
Objectives  Classification  Tool 
(LOC  Tool)  which  could  use 
selected  behavior  codes  to 
create  learning  objective 
hierarchies.  NTSC  has  another 
DRC  developed  tool  for  Media 
selection,  that  could  use 
selected  elements  of  the 
video/audio  data  base.  DRC  is 
currently  developing  a  Course 
Outline  Analysis  Tool  which 
could  use  selected  elements  to 
help  select  all  the  precursor, 
supporting,  and  development 
behaviors  used  for  structuring 
courseware.  Using  these  or 
similar  tools  (see  Figure  17), 
the  process  could  isolate 
instructional  segments  from  the 
video/audio  data  base.  These 
segments  would  be  assigned  to  a 
given  audiovisual  media,  such  as 
CBT,  and  then  organized  into  an 
instructional  sequence. 
Video/audio  data  base  segments 
and  a  requirements  list  would  be 
delivered  for  computer  based 
storyboarding  by  development 
personnel ,  and  prototype 
courseware  design  would  be 
underway  while  analysis 
continues  for  the  remaining 
instructional  requirements. 


Summary 

Until  recently,  traditional 
linear  text -based  procedures 
have  served  the  training 
community  well .  The  push  and 
pull  of  interactivity,  as  well 
as  cost,  have  driven  us  to  look 
for  more  efficient  and  effective 
processes.  (See  Figure  18) 
Application  of  interactive 
technology  to  the  data 
collection  and  analysis,  and  to 
the  development  of  training 
products  will  provide  the 
efficiency  required.  Time  for 
development  will  be  reduced,  man 
hours  per  hour  of  instruction 
will  be  reduced,  analysis 
results  will  be  more  consistent 
and  traceable  to  the  training 
requirements,  and  most 
importantly,  interactive 
technology  permits  comprehensive 
analysis  of  the  complex 
operational  environment  along 
with  the  more  accurate 
representation  of  that 
environment  in  the  instructional 
setting. 
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A  FRAMEWORK  FOR  ISO:  THE  TRAINING  ENTERPRISE  MODEL 


Robert  F.  Bachert,  Tenny  A.  Lindholm,  Ken  Evers 

Agencies  within  the  Department  of  Defense,  Industry,  and  academia,  presently  use 
variations  of  Systems  Approaches  to  Training  (SAT)  to  develop  original  training 
systems.  A  myriad  of  task  analysis  methods  are  used  in  the  necessary  requirements 
analysis.  After  initial  development,  it  appears  that  components  or  subsystems  of 
training  systems  are  developed  in  isolation,  with  little  or  no  consideration  as  to 
interfaces  or  interactions  with  other  components.  Technology  is  continually  presenting 
the  challenges  of  systems  upgrades  and,  hence,  modifications  to  training.  Technology 
may  change  the  approaches  to  training.  That  is,  all  aspects  of  training  tend  to  be 
evolutionary  or  changing.  The  Air  Force  is  developing  methodology  for  the  definition 
and  design  of  "total"  training  systems/enterprises.  The  methodology  is  based  on  the 
concepts  of  the  systems  approach  and  adaptive  evolutionary  systems.  This  paper 
discusses  these  concepts,  the  methodology,  and  their  applications  to  the  life  cycle 
phases  of  training  systems.  In  particular,  the  training  enterprise  model  which  serves 
as  a  framework  for  the  ISD  process  is  discussed,  with  ''mphasis  on  the  management, 
planning.  Total  Quality  Management  (TQM),  system  evolution  aspects  in 
relationships  to  ISD.  The  approaches  b«'ir.~  developed  are  generic  and  applicable  to 
the  development  of  any  training  system. 
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A  FRAMEWORK  FOR  ISO:  THE  TRAINING  ENTERPRISE  MODEL 


Robert  F.  Bachert,  Tenny  A.  Lindholm,  Ken  Evers 


Introduction  The  Air  Force  is 
developing  methodology,  techniques, 
and  tools  for  the  definition  and  design 
of  "total"  training  systems/enterprises. 
The  methodology  is  based  on  the 
concepts  of  the  systems  approach  and 
adaptive  evolutionary  systems.  These 
concepts  are  discussed  elsewhere 
[1,6,13,15,16,17]  as  to  the  application 
for  systems  analysis,  research,  design, 
and  evaluation.  For  purposes  here,  a 
system  is  defined  as: 

One  or  more  humans, 
possibly  aided  by  tools 
and/or  machines, 
working  to  solve  a 
problem  or  achieve  a 
goal. 

The  systems  approach  is  to  combine 
selection  and  training  with  human 
engineering  and  machine  capabilities 
to  achieve  cost  effective,  efficient 
systems  with  optimal  performance.  This 
approach  considers  the  total  system  as 
well  as  all  its  components  in  a  top- 
down,  hierarchic  manner.  Suggestions 
have  previously  been  made  [17]  "that 
training  equipment  can  be  significantly 
improved  in  both  design  and 
employment  through  adapting  a 
systems  approach  which  will  integrate 
engineering  and  behavioral  science 
data."  Advantages  of  this  approach  for 
training  systems  development  include; 

1.  component  impact  on 
total  system 
performance  can  be 
evaluated 

2.  one  disciplinary  aspect 
is  not  overemphasized 
at  the  expense  of 
others 

3.  decisions  can  be 
based  on  knowledge 


of  effects  on  system 
performance  rather 
than  a  single  activity 

4.  retrofits  and  retraining 
can  be  minimized 

5.  better  rapport  among 
ail  life  cycle  team 
members 

6.  activity/function 
interrelations  can  be 
considered  at  each 
stage  of  the 
development/life  cycle. 

When  discussing  the  concept  and 
definition  of  systems,  it  is  particularly 
important  to  observe  that  "There  is  no 
such  thing  as  an  independent  objective 
perception  of  a  system"  [13].  Lendaris 
discusses  in  detail  the  definition  of 
systems,  the  systems  approach, 
perception  of  systems,  and  systems 
hierarchy.  Systems  hierarchy 
embodies  the  concept  of  suprasystems 
and  subsystems,  i.e.,  every  system  is 
contained  within  a  larger  system,  and 
every  system  contains  a  subsystem 
(with  obvious  limits  in  each  direction). 
Understanding  these  concepts  is  vital  to 
the  application  of  the  systems  approach 
to  training  systems  design  [10].  It  can 
be  postulated  that  the  lack  of 
understanding  has  led  to  the  quagmire 
of  task  analysis  methods.  Methodology 
and  techniques  discussed  in  this  paper 
have  been  developed  to  accommodate 
the  above  concepts  and  their 
applications  to  the  planning  and  design 
of  training  systems.  That  is,  training 
systems  development  must  begin  at  a 
high-enough  level  within  the  enterprise 
to  encompass  all  relevant  training 
requirements.  Deciding  exactly  how 
high  remains  the  biggest  challenge: 


however,  this  decision  must  be  made 
prior  to  systems  design. 

Just  as  the  definition  of  system  can 
vary,  so  also  the  definition  of  training 
system.  Hays  [10]  says  "A  training 
system  is  the  planned  interaction  of 
people,  materials,  and  techniques, 
which  has  the  goal  of  improved 
performance  as  measured  by 
established  criteria  on  the  job."  This 
definition  is  general  enough  to  include 
training  systems  of  all  sizes  and  is 
certainly  compatible  with  the  systems 
definition  previously  given.  It  is  not 
quite  clear  whether  improved 
performance  means  added  capability 
for  students  who  will  be  new  to  a  job  or 
increased  systems  efficiency  and 
effectiveness.  Variables  impacting 
human  performance  are  discussed 
below. 

For  purposes  here  an  enterprise  is 
defined  as;  a  system  that  produces 
products  to  achieve,  goals  and 
objectives,  is  influenced  by  customers, 
and  controlled  by  its  suprasystem. 
Thus,  an  enterprise  may  be  a  system,  or 
training  system,  consisting  of  only 
humans,  or  it  may  be  a  human/machine 
system.  Size  and  complexity  may  vary 
from  a  single  human  (e.g.,  one  person 
company)  to  a  multinational 
organization  (e.g.,  a  Fortune  500 
corporation).  Humans  build  enterprises 
to  solve  problems  related  to  larger 
(supra)  enterprises  for  use  as  tools  in 
these  larger  enterprises.  The  attempt 
has  been  to  make  that  definition 
consistent  with: 

1.  the  concepts  of  Total 
Quality  Management 
(TQM),  which  will  be 
discussed  in  more 
detail  below 

2.  the  systems  approach 


3.  the  definition  of 
training 

systems/enterprises. 

An  objective  is  to  use  the  enterprise 
concept  to  orient  the  systems  concept 
practitioners  towards  humans  centered, 
adaptive,  self-learning,  and 
evolutionary  type  systems,  in  particular, 
for  this  paper,  training  systems. 

From  the  above  definitions,  a  training 
enterprise  would  be  one  or  more 
humans,  using  tools  and/or  machines 
(media),  to  produce  humans  (or 
animals,  etc.)  trained  for  specific 
objectives  internal  to  or  external  to  the 
enterprise  as  determined  by  the  user  or 
customer.  As  will  be  discussed  below, 
the  Instructional  System  Development 
(ISD)  process  is  a  model  (or  set  of 
guidelines)  to  be  used  by  the  enterprise 
to  structure  its  activities  for 
accomplishing  the  training. 

Background  Systems  approach  to 
training  (SAT)  and  its  successor  ISD 
have  been,  and  are  being,  used  to 
develop  training  systems  for  a  large 
variety  of  systems.  Hays  and  Singer 
[10]  provide  a  good  history  and 
evaluation  of  both  approaches.  The 
application  of  these  procedures  have 
been  difficult  for  some  practitioners  and 
the  reasons  for  this  probably  depend  on 
a  number  of  variables.  Following  are 
extracts  from  references  [10,12]  that 
provide  a  flavor  of  their 
viewpoint/perception  for  suggested 
problems  and  proposed  solutions. 

"A  third  area  of  concern  is 
improving  the 
comprehensiveness, 
accuracy,  and  resolution 
of  system  function  models 
for  both  normal  and 
abnormal  modes  of 
operation."  [12] 
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"Task  analysis  and 
instructional  systems 
development  methods  are 
useful  to  determine  what 
is  to  be  trained  but  not 
explicitly  to  determine 
what  is  required  to 
conduct  training."  [12] 

"Improvements  are 
needed  in  task  analysis 
techniques  and 
associated  inferential 
methodologies."  [12] 

"A  long-term  research 
program  should  be 
initiated  to  provide  tools 
for  task  analysis 
encompassing  the 
description  of  operational 
tasks,  the  decomposition 
of  task  description  into 
parameters  and  attributes 
relevant  to  the  design 
objectives,  and  synthesis 
of  the  analytical  data  into 
requirements  or 
specifications  for  system 
design."  [12] 

"Recent  research 
supporting  SAT  and  its 
offspring,  ISD,  reflects  a 
return  to  the  1950's  belief 
that  the  development  of 
training  is  a  complex 
dynamic  problem 
requiring  the  techniques 
of  systems  analysis."  [1 0] 

"A  true  systems  approach 
is  necessary  to  design 
effective  training 
programs,  but  rigid 
proceduralized  methods 
should  not  be  applied  in 
every  case."  [10] 


"What  is  necessary  to 
make  systems 
approaches  to  training 
work  is  guidance  that 
helps  training  developers 
conduct  the  procedures 
recommended  by  ISD. 

Such  guidance  must  be 
based  on  a  valid  model  of 
training  systems."  [10] 

This  paper  discusses  a  top-down 
systems  approach  and  its  application  to 
the  planning,  design,  and  systems 
engineering  of  evolutionary  training 
enterprises.  In  particular,  a  generic 
training  enterprise  model  containing  the 
ISD  process  and  its  utilization  as  a 
framework  for  training  system  program 
development  will  be  considered. 

The  Training  Enterprise  There  are  two 
types  of  training  enterprises.  Each  has 
as  its  purpose  the  goal  of  training 
humans  as  given  by  the  above 
definition.  However,  one  can  be 
independent  of  the  enterprise  using  its 
services  and  one  may  be  contained 
within  the  customer’s  organization.  For 
example,  a  university  is  independent  of 
organizations  that  hires  its  graduates, 
while  General  Motors  has  traditionally 
trained  its  own  employees  through  the 
GM  Institute.  There  are  both  kinds  of 
training  enterprises  within  the 
government.  Of  course,  an  enterprise 
can  be  both  independent  of  its 
customer  yet  contained  within  a 
customer  parent  organization.  For 
example,  the  Air  Training  Command 
within  the  United  States  Air  Force. 
Hence,  when  defining  an  enterprise  it 
becomes  important  as  to  the  scope  of 
interest  and  perception. 

Training  systems  developed  for  the 
Department  of  Defense  can  vary  in  size 
from  maintenance  training  systems  to 
large,  complex,  pilot  training  systems. 


The  Air  Force,  under  its  Specialized 
Undergraduate  Pilot  Training  System 
program  has  been  developing  a 
Management,  Planning,  Analysis,  and 
Definition  System  (MPADS)  to  be  used 
for  the  development  of  adaptive, 
evolutionary  training  enterprises  of  all 
sizes  and  complexities.  Portions  of  this 
program  include  the  development  of: 

1.  a  framework  for  enterprise 
system  analysis  (3) 

2.  an  enterprise  model  for 
training  systems  (7) 

3.  a  model  of  the  ISD  process 
(9). 

Figure  1  represents  a  top-level 
definition  and  bounding  of  a  training 
enterprise.  Figure  2  represent  the  next 
level  decomposition  of  Figure  1.  This 
model  indicates  the  functions  by  which 
an  adaptive,  evolutionary  training 
enterprise  accomplishes  its  goal  and 
produces  its  products.  This  model  is 
described  in  more  detail  elsewhere  (7). 

The  IDEAL  Methodology  IDEAL 
(Integrated  Design,  Engineer,  and 
Analysis  Languages)  is  a 
comprehensive  methodology  for 
describing,  developing,  and  analyzing 
systems  (7).  Figure  3  shows  the 
analytic  framework  presented  by 
Bachert,  et  ai,  (3),  and  the  application  of 
IDEAL  on  the  Depth  dimension  and 
MPADS  on  the  Breadth  dimension. 

IDEAL  has  proven  to  be  a  unique, 
integrated  approach  that  utilizes  a  top- 
down,  structured  technique  to  define 
and  document  the  system  of  interest:  a 
knowledge  engineering  technique  to 
collect  and  organize  system  descriptive 
and  task  information;  a  rapid 
prototyping  technique  to  perform 
preliminary  systems  performance 
analysis;  and  a  sophisticated  simulation 
technique  to  perform  in-depth  system 
performance  analysis. 


Figure  3.  MPADS  Analytical 

Framework:  Planner, 
Developer  Roles 


IDEAL  has  proven  to  be  an  effective 
approach  through  the  integration  of 
proven  techniques  that  have  been 
developed  and  proven  as  stand  alone 
methods.  Among  these  are  IDEF(O), 
IDEF(1),  SADT,  and  SAINT.  )DEF(0) 
and  SADT  are  techniques  for 
graphically  describing  a  system  from  a 
functional  and  informational 
perspective.  The  SAINT  simulation 
language  provides  a  graphical 
representation  of  the  dynamic 
performance  of  man-machine  or 
generic  systems.  IDEAL  integrates  the 
static  and  dynamic  representations 
through  the  use  of  a  Performance  Data 
Base. 

IDEF(O)  originated  as  a  software 
requirements  engineering  technique.  It 
has  subsequently  been  extended  to 
other  applications  where  a  top-down 
decomposition  approach  was  needed 
to  describe  a  system  in  order  to 
examine  how  the  system  does  or 
should  function.  The  principal  goal  of 
IDEF(O)  is  to  provide  a  structured 
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Figure  1.  A-0  Diagram  Model  of  Training  Enterprise 


Figure  2.  AO  Diagram  of  IDEF(O)  Model  of  Training  Enterprise 
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approach  for  breaking  a  complex 
system  into  more  elemental 
components  that  are  simpler  to  deal 
with.  This  is  the  basis  for  contemporary 
systems  engineering  practices  where  a 
multi-disciplinary  team  must  be 
employed  to  design,  develop,  and 
produce  a  product. 

An  IDEF(O)  model  consists  of  an 
integrated  set  of  diagrams  that  define 
the  system  boundaries  and  the  system 
structure  in  a  top-down  manner. 
Diagrams  consist  of  boxes  (defining 
system  function/activities/tasks)  and  of 
data  arrows  (defining  relationships  or 
information  flow  among  the  functions). 
The  information  associated  with  each 
activity  is  comprised  of  inputs  to  the 
activity,  outputs  produced  by  the 
activity,  controls  which  represent  the 
resources  (people,  equipment, 
software,  hardware,  etc.)  which  are 
responsible  for  the  transformation.  As 
illustrated  in  Figure  4,  through  the 
development  of  an  IDEF(O)  model,  a 
system  understanding  is  gained  in  a 
gradual,  controlled  manner  through  a 
graphic  representation. 


Figure  4.  IDEF(O)  Functional  Model 


The  rDEF(O)  and  SAINT  techniques  are 
closely  related  in  that  the  activities 
defined  by  IDEF(O)  are  the  same  as 
those  represented  in  SAINT.  Also,  the 
data  relationships  among  the  activities 
in  both  techniques  are  the  same.  The 
primary  difference  between  the  two 
techniques  is  that  IDEF(O)  is  a  static 
representation  of  the  system  whereas 
SAINT  provides  a  dynamic 
representation  of  the  system.  To  form 
the  link  between  the  static  and  dynamic 
representations,  the  Performance  Data 
Base  (PDB)  was  formulated.  The  PDB 
is  filled  by  specifying  dynamic 
information  for  each  sub-activity 
represented  in  the  IDEF(O)  model  so  as 
to  define  the  performance 
characteristics  of  each  activity  along 
with  the  performance  relationships 
among  the  activities.  The  dynamic 
representation  of  the  system  is  then 
developed  through  the  SAINT 
language  by  forming  the  basic  network 
through  the  IDEF(O)  model  onto  which 
the  dynamic  information  from  the  PDB 
is  integrated. 

Human  Performance  and  Systems 
Design  If  the  purpose  of  training  is  to 
improve  performance,  then  human 
performance  variables  must  be 
considered  when  designing  systems 
and  their  corresponding  training 
systems.  Holt  and  Stevenson  (11) 
stated  that; 

"Systems  designers,  when 
confronted  with 
widespread  human 
performance  problems  in 
their  systems,  search  for  a 
broader  design  approach. 

They  sometimes  believe 
that  training  equates  to 
satisfactory  human 
performance.  There  are  at 
least  two  flaws  in  such  an 
assumption.  First,  human 
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performance  in  systems  is 
the  complex  interactions 
of  many  variables,  one  of 
which  is  training...  (The 
other  variables  are 
design,  human  machine 
interface,  information 
transfer,  environment, 
personal  factors, 
supervision,  and 
documentation.)" 

These  variable  are  not  independent. 
Hence,  when  determining  training 
needs  the  following  issues  must  be 
considered: 

1)  Does  the  training 
system  need  to  be 
evolutionary  to 
accommodate  an 
evolutionary  system? 

2)  Where  in  the  design 
process  should 
training  requirements 
be  determined? 

3)  What  trade-off 
analysis/questions  for 
human  performance 
variables  should  be 
done/answered? 

4)  What 

resources/systems 
analysis  tools  are 
available  to  answer 
issues  1  thru  3? 

Rerujirements  The  training  enterprise 
framework  and  analytic  framework 
discussed  above  are  used  with 
appropriate  requirements  to  design  and 
develop  an  operational  training  system. 
Figure  1  and  its  textual  explanation 
define  what  the  training  enterprise's 
mission  is,  i.e.,  what  the  enterprise  is  to 
accomplish,  its  purposes  and 


objectives.  DeGreene  (5)  gives  the 
following  definition  of  a  requirement. 

"A  statement  of  an 
obligation  the  system  must 
fulfill  to  affect  the  mission. 
Requirements  are 
expressed  first  in 
qualitative  terms  and 
progressively  in 
quantitative  performance 
terms  relative  to  some 
criterion(ia).  They  further 
delineate  the  system 
mission." 

Top  level  requirements  for  the  training 
enterprise  shown  are: 

1.  Be  an  adaptive, 
evolutionary,  self 
designing  system. 

2.  Incorporate  TQM. 

3.  Perform  top-down  long 

term  and  short  term 
level  planning  for 
enterprise  and  product 
change  and 

evaluation. 

4.  Perform  requirements 
analysis  for  customer 
needs. 

5.  Incorporate  ISD  and 
simulation  capabilities. 

6.  Be  human 

performance  and 
system  effectiveness 
oriented. 

DeGreen  gives  the  following  definition 
of  function. 

"A  general  means  or 
action  by  which  the 
system  fulfills  its 
requirements.  Functions 
are  usually  expressed  in 
verb  form.  They  are  the 
expression  of  the  how  of 
the  system.  They  are 


conceived  apart  from 
implementation  by  men 
and/or  by  machines:  in 
practice,  they  are  usually 
expressed  along  with 
machine  design 
implications." 

The  definitions  given  by  DeGreene  are 
basic  terms  in  systems  hierarchy  and 
are  equivalent  to  the  IDEAL 
methodology  and  its  application  to 
systems  requirements  and  design.  As 
the  enterprise  is  progressively 
decomposed  into  lower  levels  of  detail, 
requirements  can  also  be  expressed 
progressively  in  quantitative 
performance  terms.  In  the  design 
phase  the  mechanisms  of  humans 
and/or  machines  are  allocated:  the 
requirements  for  these  mechanisms 
have  been  predetermined. 
Requirements  addressed  by  the 
MPADS  framework  models  and  IDEAL 
include: 

Functions, 

Data,  Information,  Knowledge, 
Internal  and  External  Inter.aces, 
Human/Machine  Allocation, 
Operations, 

Performance. 

Logistics, 

Environment. 

With  the  top-down,  hierarchic 
decomposition,  the  various  levels  of 
requirements  are  well  documented  and 
mappable  to  the  corresponding 
function,  data,  and  management  levels. 
Depending  on  the  size  and  complexity 
of  the  function,  different  size  teams  may 
be  needed  to  accomplish  the  activity  or 
task.  Thus,  requirements  for  team 
design  and  training  can  be  determined. 
Also  it  is  possible  to  study  and  evaluate 
team,  as  well  as  individual, 
performance  in  relation  to  the  eight 
variables  previously  discussed.  This  is 
a  top-down  task  analysis. 


Requirements  for  the  training  enterprise 
were  that  it  perform  requirements 
analysis  for  customer  needs  and 
incorporate  the  ISD  and  simulation 
capabilities.  The  methodology  and 
techniques,  discussed  above  to  satisfy 
these  requirements,  are  the  same  that 
should  be  used  to  perform  customer 
requirements,  task  analysis  for  ISD,  and 
dynamic  analysis  via  simulation,  i.e., 
system  analysis.  Figure  5  is  an  IDEF(O) 
node  tree  showing  a  further 
decomposition  of  the  functions  shown 
in  Figures  1  and  2.  It  shows  where, 
from  a  functional  perspective,  the 
various  requirements  are  implemented. 
Note  the  ISD  process  indicated  by  the 
shaded  areas.  IDEF(O)  diagrams  (sub¬ 
models)  exist  for  each  branch  of  the 
node  tree.  The  validation  and  further 
development  of  this  generic  model  is  in 
progress  as  part  of  the  MPADS  and 
SUPT  programs. 

Total  Quality  Management  fTQM^  The 
relationships  between  management, 
planning,  problem  solving,  training 
systems  development  methodologies, 
systems  analysis  and  modeling,  and 
TQM  must  be  well  defined  for  TQM  to 
be  effective.  This  can  be  facilitated 
through  the  methodologies  and  tools 
described  above  to  support  the 
application  of  TQM  to  the  training 
enterprise  [1].  It  is  necessary  to  better 
understand  an  enterprise's  structures, 
functions  and  performance  in  the 
context  of  a  total/integrated  operation 
that  is  continually  changing.  From  this 
knowledgebase  the  training  enterprise 
can  set  objectives,  define  strategies, 
and  plan  an  effective  application  of  a 
TQM  process. 

Using  the  framework  components  as  a 
guide  (the  enterprise  model  is  a 
reference  model  to  be  used  with  other 
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Figure  5.  Training  Enterprise  Model  Node  Tree 

Shaded  function  block  correspond  to  ISD  Process 


framework  models)  the  planning  of 
TQM  is  accomplished  as  follows: 

•  All  external  and  interna' 
inputs  are  identified  and 
it  is  possible  to  trace 
what  functions  or 
components  of  the 
enterprise  are  impacted. 

For  each  function  the 
quality  a  n  d 
specifications  of  impacts 
can  be  evaluated. 

•  The  capabilities,  etc.,  of 
human  and/or  machine 
mechanisms  necessary 
to  accomplish  the  task 
can  be  planned. 

•  Controls  are  designed 
using  the  requirements 
of  the  process  and 
mechanisms  allocated. 

•  System  or  process 
related  problems  are 


addressed  via  modeling 
and  simulation. 

Since  the  framework  models  are  top- 
down  and  each  level  of  decomposition 
provides  a  greater  level  of  detail,  it  is 
possible  to  plan  for  TQM  considerations 
for  various  levels  of  organizational  and 
functional  structure.  In  training  systems 
quality  equates  to  human/system 
performance  issues  and  the  validity  of 
simulations/simulators. 

Summary  This  paper  has  discussed 
portions  of  the  Air  Force  program  to 
develop  a  planning  system  (MPADS) 
for  the  planning,  analysis,  design, 
development,  test  and  evaluation,  and 
the  evolution  of  training  enterprises. 
The  concepts  and  methodology 
embodied  in  this  system  is  the  top- 
down  systems  approach.  The  basis  of 
MPADS  is  a  framework,  or  reference 
model,  for  describing,  analyzing, 
evaluating,  and  designing  large 
complex  training  enterprises.  It  is  used 
as  the  "framework"  for  systems 
engineering,  component  development. 
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product  development,  enterprise 
design/operation,  and  enterprise 
management.  Models  derived  via  the 
framework  (e.g.,  the  training  enteiprise 
model)  as  part  of  MPADS  will  provide: 

1.  a  clearer  picture  of 
what  the  training 
enterprise  is  doing  and 
where  it  is  going 

2.  scenario  analysis  that 
can  be  used  to  answer 
"what  if  questions 

3.  reduced  analysis  time 
and  elimination  of  the 
intuitive  approach 

4.  an  accurate  picture  of 
decision 
consequences 

5.  a  "corporate  memor 
database 

6.  rapid  ans’vr  ^  to 
questions  a->out  the 
current  status  of 
organ'^ations. 

The  planning  system  is  being  designed 
to  consider  the  issues  of  defining 
problens/needs,  planning  for  large 
complex  training  systems,  and 
designing  the  appropriate 
organizational  infrastructure  necessary 
for  an  evolutionary  adaptive  system. 
The  methodology,  techniques,  and 
[ools  discussed  in  this 
paper  are  applicable  to  all 
systems/enterprises. 
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A  TRAINING  DEVELOPMENT  ENVIRONMENT  IN  HYPERTEXT 


C.G.  Thronesbery,  Ph.D. 

Ron  W.  Rhoads 

The  development  of  training  systems  for  the  military  requires  the 
management  of  large  amounts  of  information.  The  developer  must  ensure 
that  the  lesson  being  developed  is  consistent  with  the  task  being  performed, 
the  individual  training  plan,  collective  training  plans,  doctrinal  sources,  and 
other  student  reference  sources.  Often,  the  number  of  resource  materials 
available  to  the  developer  is  so  large  that  it  is  nearly  impossible  for  the 
developer  to  give  due  consideration  to  every  source.  The  result  is  that  the 
lesson  is  less  complete  and  less  powerful  than  if  all  sources  had  been 
considered.  Furthermore,  once  a  lesson  has  been  developed,  the  developer 
needs  to  maintain  records  of  the  sources  of  his  information  in  the  event  that 
a  particular  point  is  challenged.  The  information  to  be  managed  consists  of 
text  and  graphics  of  varying  size  and  complexity.  Such  information  has  not 
been  addressed  well  by  traditional  computer  software  packages  like  word 
processors,  data  base  management  systems,  and  spreadsheets.  Hypertext  is 
designed  to  organize  this  type  of  information,  but  is  often  difficult  to 
provide  the  reader  with  a  clear  enough  structure  so  that  he  remains  oriented 
and  capable  of  performing  a  directed  search. 

A  structure  for  a  hypertext  system  is  described  which  will  serve  as  a  training 
development  environment.  Specific  screen  designs  and  hypertext  structures 
are  presented  which  support  training  development.  Assuming  arrangements 
can  be  made  for  a  large-display  Macintosh  system,  a  demonstration  of  a 
sample  training  development  hypertext  structure  is  anticipated. 

C.G.  THRONESBERY.  Dr.  Thronesbery  joined  The  MITRE  Corporation  as  a  Member  of 
the  Technical  Staff  in  September  of  1990,  in  support  of  NASA  at  Johnson  Space  Center. 
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University  of  Oklahoma. 
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Hypertext  is  often  useful  for  managing 
large  amounts  of  heterogeneous 
information.  One  task  which  has 
extensive  information  requirements  is 
training  development.  Consequendy,  a 
sample  hypertext  system  is  presented 
which  suppons  training  development 

When  a  large  volume  of  heterogeneous 
data  must  be  searched  to  find  the 
information  of  interest,  hypertext  is  often 
the  information  management  technology 
of  choice  (Shneiderman,  1989).  Unlike 
database  management  applications,  which 
also  allow  large  amounts  of  information 
to  be  searched,  hypertext  systems  do  not 
require  a  rigid  record  structure.  In  fact, 
the  data  can  be  so  heterogeneous  as  to 
include  computer  graphics,  video  disk 
images,  sound,  etc.  Hence  the  emerging 
term  "hypermedia." 

In  their  analysis  of  hypenext,  Foskett 
(1990),  McAleese  (1989),  Shneiderman 
(1989X  ar  d  van  Dam  (1988)  indicate  that 
^though  tne  data  may  be  heterogeneous, 
structure  is  required  of  hypertext  data 
bases  to  promote  readabdiiy. 

Thronestery  and  Rhoads  (1990a)  indicate 
that  hypertext  development  environments 
which  expedite  the  construction  of 
readable  hypertext  structures  will 
probably  be  available  at  a  future  date. 
However,  they  indicate  that  before  these 
environments  can  be  constructed,  a 
number  of  special-purpose  hypertext 
structures  must  first  be  built. 
Consequently,  the  present  paper 
describes  how  to  construct  a  hypenext 
document  system  which  will  suppon 
training  development.  Hopefully,  after  a 
number  of  such  systems  have  bwn 
described,  generic  properties  of  readable 
hypenext  structures  will  become  evident 


and  can  be  incorporated  into  future 
hypenext  construction  environments. 

In  keeping  with  the  theme  of  this  year's 
conference,  the  current  paper  shows  how 
to  use  the  technology  of  hypertext  for 
training  development.  Specific  screen 
designs  and  hypenext  structures  for  a 
sample  hypenext  document  system  are 
described  which  will  suppon  the  specific 
needs  of  training  development. 

I.  Training  Development  Requirements. 

The  developer  of  military  training  must 
manage  a  tremendous  amount  of 
information.  He  must  consult  sources  of 
doctrine,  task  analyses,  mission 
statements,  training  plans,  and  other 
training  modules.  The  content  of  the 
lesson  he  develops  must  be  consistent 
with  doctrinal  sources,  which  means  that 
he  must  consult  the  appropriate  Field 
Manuals  (I^s)  and  Student  Texts  (STs). 
The  selectiOT  of  objectives  for  the  lesson 
must  be  consistent  with  individual  and 
collective  training  plans,  e.g..  Mission 
Training  Plans  (MTPs)  and  Army 
Training  Evaluation  Plans  (ARTCPs).  In 
the  event  that  the  training  plan  is  not 
explicit  enough  to  make  a  decision  about 
selecting  a  particular  objective,  the 
training  developer  might  need  to  consult  a 
Mission  Essential  Task  List  (METL)  and 
a  Task  Analysis  Information  Sheet 
(TAIS)  in  oi^r  to  determine  critical  tasks 
for  training.  Finally,  the  developer  must 
ensure  that  the  current  training  itKxlule  is 
consistent  in  format,  content,  and 
presentation  sequence  with  the  other 
lessons  in  the  course.  This  latter  problem 
is  compounded  if  the  developer  is  part  of 
a  team  in  which  each  member  is 
developing  a  different  portion  of  the 
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course.  Because  of  this  enormous 
requirement  for  information  coordination, 
training  developers  often  have  several 
opened  resource  materials  covering  the 
lops  of  their  desks,  their  chairs,  their 
friends'  desks,  their  friends'  chairs,  etc. 

Moreover,  the  training  developer  is  often 
expected  to  defend  individual  points 
covered  in  a  lesson  after  a  considerable 
amount  of  time  has  passed.  He  must 
identify  the  doctrinal  source  in  which  an 
exercise  or  objective  is  grounded.  If  he 
has  not  recorded  this  information 
rigorously,  he  will  be  required  to  repeat 
the  entire  process  of  researching  the 
lesson  module. 

n.  Difficulties  in  Meeting  Information 
Requirements. 

Because  there  are  so  many  potential 
sources  of  information  to  coordinate,  the 
training  developer  typically  spends  a  lot 
of  time  consulting  these  sources  and, 
despite  his  diligence,  still  overlooks  some 
of  them.  For  instance,  because  the 
developer  has  forgotten  that  a  course  he 
developed  two  years  ago  has  a  «'inilar 
training  module  to  the  one  he  is  currently 
developing,  he  might  miss  the 
opportunity  to  save  himself  a  lot  of  work. 
Because  he  has  doesn't  have  easy  access 
to  his  co-worker's  work-in-progress,  he 
might  miss  an  opportunity  to  coordinate 
an  effective  presentation  style  with  the 
developer  of  a  related  training  module. 
While  he  may  work  primarily  with  one  or 
two  doctrinal  sources,  he  may  overlook 
other  manuals  with  small,  yet  valuable, 
sections  related  to  the  current  lesson 
module.  In  other  cases,  the  training 
developer  might  know  of  a  source  and 
intend  to  consult  it,  but  be  unable  to 
locate  it  Finally,  because  there  are  so 
many  potential  sources,  it  is  often 
difficult  to  place  the  relevant  ponions  of 
each  source  in  proximity  to  one  another 
so  that  they  can  be  directly  compared.  To 


summarize,  the  training  developer  should 
consult  a  large  number  of  references  to 
develop  a  high  quality  lesson  module,  he 
spends  a  lot  of  time  doing  so,  and  he  still 
misses  a  number  of  them.  Thus,  the 
efficiency  and  quality  of  lesson 
development  suffer. 

III.  How  Hypertext  Can  Help  to  Meet 
Requirements. 

Hypertext  is  a  technology  which  can 
enable  its  readers  to  locate  specific 
information  within  a  large  body  of  widely 
varying  types  of  information.  Hypertext 
is  a  way  of  storing  and  presenting 
information,  typically  with  the  aid  of  a 
computer.  In  the  typical  hypertext 
system,  a  passage  is  presented  which 
contains  one  or  more  highlighted  terms. 
The  reader  may  designate  any  highlighted 
term  and  subsequently  see  another 
passage  which  contains  additional 
information  concerning  the  designated 
term.  Each  new  passage  wiU,  in  turn, 
have  highlighted  words  or  phrases.  For  a 
more  complete  definition  and  an  overview 
of  a  number  of  existing  hypertext 
systems,  Conklin  (1987)  has  written  a 
very  useful  review.  Also,  Thompson  and 
Thompson  (1987)  have  written  a  creative 
article  to  demonstrate  the  nature  of 
hypertext  to  people  who  do  not  have 
hypertext  systems. 

By  using  a  hypertext  system  containing 
the  doctrinal  sources,  similar  to  diat 
developed  by  Thronesbery  and  Rhoads 
(1990b),  the  training  developer  could 
locate  information  Ite  might  otherwise 
have  overlooked,  either  because  he  was 
unaware  of  its  existence  or  because  he 
had  simply  forgotten  about  it  In 
addition,  if  he  and  his  co-workers  were 
using  a  shared  hypertext  data  base,  he 
could  look  at  the  portions  of  their  work- 
in-progress  which  are  related  to  his 
current  lesson  module.  He  would  also  be 
able  to  compare  the  modules  in  his  course 


131 


Training  Development  Environment  in  Hypertext  -  Thronesbery  &  Rhoads 


to  his  training  plan.  Upon  detecting 
ambiguities  in  the  plan,  he  could  consult 
sources  like  the  METL  and  the  task 
analysis  to  determine  appropriate  training 
objectives  and  lesson  modules  for  the 
current  course.  Furthermore,  if  the 
hypertext  system  were  implemented  on  a 
large-screen  workstation  with  windowing 
capability,  he  could  examine  the 
information  from  a  number  of  sources 
simultaneously.  Finally,  since  hypertext 
''an  also  function  as  an  authoring 
environment,  it  can  be  used  to  develop 
the  training  module  using  hypertext  links 
to  identify  the  references  upon  which  the 
objectives  are  based. 

IV.  Required  Hypertext  Properties. 

While  hypertext  promises  a  tremendous 
increase  in  efficiency,  it  will  deliver  on 
that  promise  only  if  it  is  carefully 
structured  to  support  the  information 
requirements  of  the  training  developer. 
The  following  is  a  list  of  the  most 
important  properties  the  hypertext  system 
must  have  to  suppon  a  training  developer; 

•  allow  quick  navigation  to  familiar 
locations  within  the  hypertext 
document  system 

•  provide  cues  where  information  can 
be  found  regarding  a  particular  topic 

•  provide  easy  navigation  among 
several  sources  related  to  a  specific 
topic 

•  allow  the  simultaneous  display  of 
several  windows  of  information  so 
they  can  be  compared  to  one  another 

•  allow  several  training  developers  to 
view  one  another's  work-in-progress 

•  allow  work-in-progress  to  be 
accessed  topically 

V.  Sample  Hypertext  Training 
DeveloDment  Environment 

The  proposed  hypertext  training 
development  environment  is  a  repository 


of  doctrinal  information,  task 
information,  training  plans,  and 
developed  courses.  T^e  screen  designs 
shown  in  the  figures  are  ftom  a 
demonstratitHi  system  generated  in  Guide 
(a  hypertext  development  environment  by 
Owl  International)  on  a  large-screen 
Macintosh  computer. 

A.  Document  Structure. 

The  hypertext  document  structure  is 
illustrated  in  Figtue  1.  The  lessons  being 
developed  arc  represented  on  the  left  side 
of  the  figure  while  the  reference  sources 
are  represented  on  the  right  side.  Each 
majOT  component  of  this  hyj^rtext  system 
has  an  inherent  structure  wluch  is 
emphasized  and  used  fcH*  navigation 
within  that  component. 

The  Lesson  Table  of  Contents  (TOO  is  a 
collection  of  previously  developed 
courses  and  work-in-progress.  When  the 
reader  designates  (clicks  on)  a  course 
name,  a  schematic  of  the  lesson  modules 
within  that  course  will  unfold  below  the 
course  name,  ’^•e  schematic  shows  the 
logical  sequences  for  the  presentation  of 
the  lesson  modules.  Clicking  any  lesson 
module  within  the  schematic  will  cause 
that  nxxlule  to  be  displayed  in  a  separate 
window. 

The  lesson  module  is  illustrated  in  Figure 
2  in  greater  detail.  There  are  three 
impoitant  divisions  of  the  window 
displaying  the  lesson  module:  the  title 
box,  the  lesson  module  itself,  and  the 
icon  row. 

First,  the  title  box  is  included  to  ensure 
that  the  reader  always  has  a  clear 
identification  of  any  lesson  module  he  is 
viewing.  The  title  box  provides  not  only 
an  identification  of  the  contents  of  this 
window  (the  module  name  and  lesson 
title),  but  it  also  shows  the  overall  context 
within  which  it  fits  (the  course  title). 
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Second,  the  lesson  module,  the  lower 
portion  of  the  lesson  module,  is 
organized  hierarchically  to  expedite  the 
location  of  a  specific  objective  of  interest 
within  the  module.  It  can  be  displayed  by 
designating  the  Show  Text  icon  (from  the 
icon  row).  This  state  of  the  lesson 
module  is  illustrated  in  Figure  2.  The 
major  divisions  of  the  lesson  are  shown 
in  boldface  type.  Clicking  on  a  major 
heading  causes  the  text  and  graphics 
associated  with  that  heading  to  be 
displayed  below  the  heading.  In  Figure 
2,  the  reader  has  clicked  on 
"Introduction,"  causing  the  introductory 
paragraph  to  be  displayed  below  its 
boldfaced  heading.  Clicking  on  the 
paragraph  will  remove  it  from  the  display 
once  again.  This  feature  allows  the 
reader  to  locate  any  specific  portion  of  the 
lesson  module  quickly. 

Finally,  the  icon  row,  just  under  the  title 
box,  provides  the  reader  with  the 
capability  of  navigating  to  other  portions 
of  the  hypertext  document  system  with 
relative  ease.  The  row  of  icons  provides 
easy  navigation  to  other  portions  of  the 
hypertext  document  system.  By  clicking 
on  the  PREV  or  NEXT  arrows,  the 
developer  can  look  at  the  previous  or  next 
lessons  in  the  course.  By  clicking  on  the 
Lesson  TOC  icon,  he  can  view  the 
Lesson  Table  Of  Contents  and  course 
schematics  illustrated  in  Figure  1 .  By 
clicking  on  the  Index  icon,  the  reader  can 
view  the  topical  Index,  which  is 
described  below. 

The  compass  rose  icon  (in  the  icon  row) 
provides  quick  access  to  specific  Index 
topics.  Figure  3  illustrates  the  appearance 
of  the  lesson  module  when  the  compass 
rose  has  been  designated.  If  the  reader 
clicks  on  one  of  the  topic  names  shown  in 
Figure  3,  the  designated  topic  of  the 
Index  will  then  be  displayed  in  a  separate 
window.  Notice  that  the  compass  rose  is 


darkened  in  Fig^  3,  showing  that  it  is 
the  currently  activated  icon. 

The  Developmental  References  icon  has 
been  included  to  assist  the  developer  in 
identifying  the  authoritative  sources  upon 
which  he  based  the  materials  in  the  lesson 
module.  Figure  4  illustrates  the 
appearance  of  the  lesson  module  when 
the  Developmental  References  icon  has 
been  designated.  If  someone  should 
challenge  the  information  presented  in  the 
lesson  naodule  at  a  later  date,  the 
developer  can  click  on  the  appropriate 
entry  under  "Dev,jlopmental  References," 
causing  the  original  authoritative  source 
to  be  opened  at  the  ^propriate  passage. 
Thus,  the  developer  can  quickly  show  the 
source  of  the  information  he  has 
presented  in  the  training  module.  Notice 
in  Figure  4  that  the  Developmental 
References  icon  has  been  darkened  to 
indicate  that  it  is  the  active  icon. 

The  Index  is  included  to  provide  the 
capability  to  search  the  entire  hypertext 
document  system  topically.  The  Index  is 
accessible  from  any  window  in  the 
hypertext  document  system  by  clicking 
on  the  Index  icon.  When  the  Index  is 
fust  displayed,  the  reader  sees  all  the 
letters  of  the  alphabet.  By  designating  a 
given  letter,  he  will  see  aU  the  topics  in 
^e  Index  which  begin  with  that  letter. 

He  can  then  click  on  the  name  of  a  topic 
of  interest,  displaying  the  names  of  all  the 
locations  in  the  hypertext  system  which 
are  related  to  that  topic.  This  includes 
work-in-progress,  developed  courses, 
doctrinal  sources,  task  an^yses,  METL, 
and  training  plans.  Qicking  on  the  name 
of  a  reference  source  under  an  Index  topic 
will  cause  the  display  of  the  designated 
reference  in  a  new  window. 

With  a  large-screen  display,  the  reader 
can  place  the  Index  window  in  the  comer 
of  the  screen,  showing  the  topic  of 
interest.  He  can  then  use  the  Index  as  a 
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reading  guide,  clicking  on  each  reference 
successively.  If  there  is  a  need  to 
compare  and  contrast  sources,  their 
respective  windows  can  be  placed  side  by 
side  on  the  screen.  This  allows  the 
training  developer  to  perform  a  high  level 
of  analysis  which  is  often  difficult  under 
normal  circumstances.  It  is  especially 
impractical  when  trying  to  compare 
passages  which  occur  within  the  same 
paper-based  book.  Thus,  creative  use  of 
a  large  screen  can  improve  the  analytical 
quality  of  the  training  developer's  work. 

Other  sources  within  the  sample  hypertext 
document  base  include  task  analysis 
information  (TAIS),  training  plan 
information  (MTPs  and  ARTEPs),  and 
doctrinal  sources  (FMs  and  STs).  Each 
of  the  major  sources  shown  in  the  right- 
hand  portion  of  Figure  1  would  have  its 
own  hierarchical  table  of  contents, 
allowing  the  reader  to  navigate  quickly  to 
any  particular  passage.  The  appearance 
of  a  passage  from  these  sources  would 
resemble  the  lesson  riKxlule  shown  in 
Figure  2.  A  passage  would  have  a  title 
box,  an  icon  row,  and  paragraph 
headings.  The  title  box  would  identify 
the  passage.  The  icon  row  would  allow 
rapid  navigation  to  other  parts  of  the 
hypertext  system,  and  the  paragraph 
headings,  when  designated  by  a  mouse 
click,  will  allow  the  display  of  their 
associated  paragraphs. 

B.  Computer  Supported  Cooperative 
Work  Environment 

Because  it  can  be  a  shared  hypertext 
document  system  for  a  group  (rf  training 
developers,  the  demonstration  system  can 
be  an  instance  of  a  computer-supported 
cooperative  work  (CSCW)  environment. 
Essentially,  this  means  that  it  assists  a 
group  of  people  in  the  sharing  of 
information  and  the  development  of  a 
large  group  project.  In  this  instance,  the 
large  group  pjroject  is  a  collection  of 
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courses.  The  work-in-progress  training 
modules  are  stored  in  the  hypertext 
document  base,  along  with  the  completed 
courses. 

Sone  discipline  must  be  exercised  to 
allow  several  people  to  use  the  same  files. 
The  first  general  nile  to  observe  is  that 
while  a  file  can  have  multiple  readers,  it 
cannot  easily  have  multiple  authors.  For 
that  reason,  it  probably  makes  sense  for 
the  author  to  build  his  document  in  a 
private  work  area  and  then  place  a  copy 
of  that  work  in  a  public  area  as  major 
portions  are  finished.  For  instance, 
assuming  that  a  group  of  training 
develop)ers  was  worl^g  on  nodes  of  a 
local  area  network  (LAN),  the  file  would 
be  edited  in  a  private  file  storage  area  until 
a  major  portion  is  finished.  At  this  time, 
the  file  would  be  copied  into  a  common 
file  storage  area  of  Ae  LAN.  No  one 
would  be  allowed  to  edit  files  directly  (xi 
the  common  file  storage  area  of  the  LAN. 
Thus,  every  file  on  the  LAN  could  be 
accessed  in  a  read-only  fashion,  allowing 
several  people  to  access  them. 

The  other  general  rule  to  observe  is  that 
ctmfiguration  management  should  be 
exercised  to  ensure  that  entries  into  the 
common  reading  area  are  ready  for 
viewing  by  the  group  and  that  updates  are 
jx>sted  in  an  ord^y  fashitm.  A  lesson 
nxxlule  would  not  need  to  be  completed 
before  adding  it  to  the  common  file 
storage  area,  but  it  should  have  most  of 
the  li^  required  to  p)erform  topical 
searches.  It  should  also  have  the  icons  in 
the  icon  row  linked  to  the  Index  and 
Lesson  TOC.  This  minimal  linking 
pmvents  a  reader  of  the  file  from 
becoming  marooned  from  the  remainder 
of  the  hypertext  document  base.  For 
purposes  of  quality  control,  it  might  be 
desirable  to  have  a  single  person 
designated  to  save  a  w^-in-progress  file 
into  the  common  storage  area. 
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While  there  are  some  CSCW  projects 
which  do  not  strictly  adhere  to  these 
rules,  they  are  used  by  computer  experts 
in  an  experimental  environment 
Instances  of  large,  experimental  CSCW 
projects  appear  in  Akscyn,  McCracken, 
and  Yoder  (1988),  who  report  on  the 
KMS  system,  and  Halasz  (1988)  and 
Trigg  and  Suchman  (1989),  who  report 
on  the  NoteCards  system.  The  people 
using  these  systems  are  highly 
experienced,  knowledgeable  computer 
professionals.  Training  developers,  on 
the  other  hand,  should  not  be  expected  to 
be  as  facile  with  a  computer  as  the  people 
using  these  systems.  Also,  because  these 
systems  are  experimental,  they  have  a 
higher  tolerance  for  occasional 
difficulties.  These  rules  simplify  the 
exchange  of  work-in-process,  so  that 
difficulties  are  kept  to  a  minimum  and 
fewer  computer  skills  are  required.  For 
other  management  considerations 
concerning  the  construction  of  hypertext 
systems,  see  Rhoads  and  Thronesbery 
(1990). 

C.  Use  as  a  Training  Presentation 
Medium. 

Depending  on  the  way  in  which  the 
lesson  module  has  been  constructed  and 
the  circumstances  sunounding  the 
presentation  of  training,  the  sample 
system  could  be  used  as  a  medium  fcx* 
presenting  the  training.  The  student  can 
use  the  Lesson  TOC  to  guide  his 
navigation  from  one  lesson  module  to  the 
next.  He  can  use  the  hypertext  structures 
within  a  lesson  module  to  view  the 
materials  prepared  by  the  training 
developer.  He  can  even  take  self- tests, 
the  answers  to  which  can  be  revealed  by 
unfolding  hypertext  structures  within  the 
lesson  n^ule.  Finally,  if  he  is  having 
difficulty  with  a  particular  point,  he  can 
follow  the  relevant  reference  links  to  read 
the  doctrinal  sources  in  more  detail.  The 
paper  by  Rhoads  and  Thronesbery  in  this 


volume  describes  a  wide  variety  of 
options  for  using  hypertext  in  computer 
assisted  instruction. 

VI.  Conclusions 

Training  developers  must  manage  a  large 
amount  of  diverse  information,  and  they 
often  must  cooperate  with  a  number  of 
others  in  their  development  efforts. 

When  properly  organized,  hypertext  is 
especially  well  suited  to  support  the 
management  of  large  amounts  of  diverse 
information.  Furthermore,  with  the 
proper  safeguards,  hypertext  can  fram  the 
basis  of  a  computer-supported 
cooperative  work  environment 
Consequently,  hypertext  can  assist 
training  developers  in  managing  training 
information  and  in  cooperating  with  one 
another  to  develop  training  modules. 
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F  igure  1,  Overview  of  Hypertext  Training  Development  Environment 


introduction 

The  supported  maneuver  commander  is  responsible  for  all  fire  support 
delivered  on  surface  targets  within  his  area  of  responsibility.  Because 
of  the  increased  technology  on  the  modem  battlefield,  the  amount  of 
fire  support,  and  the  varied  types  of  fire  support  means,  the  maneuver 
commander  must  rely  on  the  senior  artilleryman  as  his  fire  support 
coordinator  To  make  coordination  of  fire  support  and  responsive 
delivery  of  these  fire  support  means  easier,  the  artilleryman  needs  to 
know  all  of  fire  support  coordinating  measures.  This  programmed  text 
will  provide  you  with  a  background  of  fire  support  coordinating 
measures. 

Student's  Reference 
Objectives: 

Instruction 
Section  I — General 

Section  II — Permissive  Fire  Support  Coordinating  Measures 
Section  III — Restrictive  Fire  Support  Coordinating  Measures 
Lesson  Test 


Figure  2.  *1110  Lesson  Module  with  the  Show  Text  icon  activated. 
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Course  Title 

Module  Name 

Lesson  Title 

Related  Index  Entries 
boundary 

coordinated  fire  line  (CFLJ 
date  time  group  (DTC?) 

fire  support  coordination  officer  (FSCODFD) 
fire  support  coordination  line  (FSCLJ 
permissive  fire  support  coordinating  measures 
restrictive  fire  support  coordinating  measures 
cone  of  fire\ 


Figure  3.  The  Lesson  Module  with  the  Related  Entries  icon  activated. 


Course  Title 
Module  Name 
Lesson  Title 


Developmental  References 

Task  Analysis  Information  Sheet 

Student  Text:  TAOISN.  Chapters.  Sections.  Survivability  Options 
AH'TSPS-lOa 

Ffi  6-50.  Chapter  4  Field  Artillery  Cannon  Battery  Defense 
Mission  Training  Plan 
Mission  Essential  Task  List 
wnte-ins  (not  a  part  of  the  system) 


Figure  4.  The  Lesson  Module  with  the  Developmental  References  icon  activated. 
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ABSTRACT 

The  Naval  Sea  Systems  Logistics  Center  Detachment  (NAVSEALOGCENDET)  tasked  the  Naval  Underwater 
Systems  Center  (NUSC)  to  form  an  authoring  system  evaluation  team.  The  evaluation  was  intended  to  identify 
current  industry  capabilities  relative  to  future  NAVSEA/NUSC  interactive  courseware  (ICW)  authoring  requirements, 
establish  realistic  criteria  for  an  authoring  system  specification,  and  evolve  a  comprehensive  authoring  system 
speciHcation  for  a  competitive  procurement.  Due  to  the  increasing  scope  of  future  ICW  development  requirements, 
it  was  determined  that  an  authoring  system  would  be  needed  that  could  accommodate  the  instructional  characteristics 
of  the  Navy  ICW,  support  the  hardware  requirements  of  the  delivery  system,  and  still  allow  an  efficient  authoring 
process  that  would  r^uce  ICW  production  costs  and  shorten  time  lines  for  a  large  volume  of  courseware.  The 
authoring  system  evaluation  team  was  composed  of  both  government  and  contractor  representatives.  The  ultimate 
objective  of  the  evaluators  was  to  develop  a  comprehensive  yet  realistic  functional  specification  for  an  authoring 
system  procurement,  rather  than  a  comparative  evaluation  of  specific  authoring  systems.  Baseline  criteria  for  an 
authoring  system  were  formulated  from  current,  conventional  authoring  system  capabilities  and  requirements  derived 
specifically  from  the  unique  development  and  hardware  requirements  of  the  Navy  ICW.  Functional  categories 
included:  interface  features,  display  features,  graphics  features,  audio/video  features,  input  features,  [n'ocessing 
features,  and  testing  features.  Evaluation  participants  were  to  produce  the  sample  lesson  using  their  authoring 
systems.  Each  evaluation  was  conducted  at  the  vender  site  over  a  period  of  two  days.  After  the  evaluation  team 
completed  its  assessments,  an  authoring  environment  specification  was  finalized,  an  RFP  drafted,  and  a  competitive 
procurement  initiated.  The  evolution  from  the  initial  baseline  criteria  to  the  final  specification  reflected  some  basic 
changes  in  philosophy  regarding  the  authoring  process.  As  a  result,  an  authoring  environment  consisting  of  flexible 
productivity  tools  and  functions  in  addition  to  the  traditional  authoring  system  was  conceived.  This  "authoring 
environment"  represents  the  next  step  in  interactive  training  production. 
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INTRODUCTION 

The  Naval  Sea  Systems  Logistics  Center 
Detachment  (NAVSEALOGCENDET)  tasked  the  Naval 
Underwater  Systems  Center  (NUSQ  to  form  an 
authoring  system  evaluation  team.  The  evaluation  was 
intended  to  identify  cuirent  industry  capabilities  relative 
to  future  NAVSEA/NUSC  interactive  courseware  (ICW) 
authoring  requirements,  establish  realistic  criteria  for  an 
authoring  system  ^)ecincation,  and  evolve  a 
comprehensive  authoring  system  specification  for  a 
competitive  procurement. 

Current  ICW  production  requirements  necessitate 
custom  development  of  course-specific  authoring  tools, 
such  as  display-capture  routines,  database  structures,  and 
courseware  executables  in  the  "C"  programming 
language.  This  represents  a  time  consuming  and 
expensive  slice  of  the  ICW  development  cycle.  Due  to 
the  itKreasing  scope  of  future  ICW  development 
requirements,  it  was  determined  that  an  authoring 
system  would  be  needed  that  could  accommodate  the 
instructional  characteristics  of  the  Navy  ICW,  support 
the  hardware  requirements  of  the  delivery  system,  and 
still  allow  an  efficient  authoring  process  that  would 
reduce  ICW  production  costs  and  shorten  time  lines  for 
a  large  volume  of  courseware. 

Instructional  features  of  the  Navy  courseware  are 
characterized  by  the  nrtorial.  demonstrate-practice, 
feature  learning,  hypothesis  testing,  and  structured 
simulation  instructional  strategies.  Delivery  system 
hardware  consists  of  custom  procured,  commercially 
available  interactive  videodisc  systems.  The  delivery 
systems  are  composed  of  a  486  based  processor,  dual 
high  resolution.  1280x1024  touch  screen  monitors; 
audio  and  video  digitizing  boards;  and  numerous 
peripheral  input  devices  arranged  in  a  cabinet  intended  to 
simulate  the  visual,  spatial,  and  tactile  characteristics  of 
an  AN/SQQ-89  sonar  console  (see  figure  1). 

The  authoring  system  evaluation  team  was 
composed  of  both  government  and  contractor 
representatives.  The  ultimate  objective  of  the 
evaluators  was  to  develop  a  comprehensive  yet  realistic 
functional  specification  for  an  authoring  system 
procurement,  rather  than  a  comparative  evaluation  of 
specific  authoring  systems.  The  team  sought  to 
determine  the  extent  to  which  current  authoring  systems 
could  meet  the  baseline  criteria  and  shape  the  authoring 
system  requirements  to  reflect  viable  expectations. 


EVALUATION  METHOD 

Baseline  criteria  for  an  authoring  system  were 
formulated  from  current,  conventional  authoring  system 
capabilities  and  requirements  derived  specifically  from 


the  unique  develqiment  and  hardware  requirements  of 
the  Navy  ICW.  Functional  categories  included; 
interface  features,  display  features,  grai^iics  features, 
audio/video  features,  input  features,  processing  features, 
and  testing  features. 

A  sample  ICW  lesson  was  then  designed  that 
incorptHated  many  of  the  critical  features  of  the  baseline 
criteria.  A  flowchart,  storyboards,  and  design 
specifications  were  formulated  for  the  lesson;  and 
sample  graphics  (both  bit  m^ped  and  vectOT),  digital 
audio  files,  and  a  DOS  executable  were  produced  to 
provide  a  realistic,  prototype  ICW  lesson.  The 
flowchart,  stcayboaids,  and  design  specifications  required 
that  the  sample  files  and  an  existing  videodisc  (provided 
by  the  evaluation  team)  be  utilized  as  part  of  the  sample 
lesson.  Evaluation  participants  were  to  p'^oduce  the 
sample  lesson  using  their  authoring  systems. 

Upon  completion  of  the  sample  lesson,  a 
Commerce  Business  Daily  ((TBD)  notice  was  drafted 
inviting  all  interested  vendors  of  authoring  systems  to 
participate  in  the  evaluation.  As  a  result,  five  authoring 
systems  were  subsequently  evaluated. 


EVALUATION  PROCESS 


Each  evaluation  was  conduced  at  the  vender  site 
over  a  period  of  two  days.  The  evaluation  team 
provide  the  flowchart,  stwyboards,  design 
specifications,  floppy  disks  with  necessary  files,  and  a 
videodisc  to  a  vendor-a^^inted  developer  for 
implementation.  The  developer  then  authored  the 
sample  lesson  while  the  evaluation  team  observed, 
askeri  questions,  and  participated  peripherally.  Each 
company  produced  different  portions  of  the  lesson  in 
different  ways,  according  to  the  characteristics  of  their 
system.  The  interaction  between  the  developer  and  the 
evaluation  team  during  lesson  production  was  frequently 
more  informative  than  anticipated.  The  evaluation  team 
often  offered  possible  solutions  to  authoring  problems 
not  previously  considered  by  the  developer.  In  addition, 
the  process  of  working  through  these  problems  with 
different  authoring  systems  also  helped  to  clarify  many 
of  the  Navy  development  requirements. 

A  key  element  of  the  evaluation  was  the 
evaluation  sheet  used  by  each  team  member.  Each 
evaluation  sheet  contained  a  list  of  production  features 
called  for  by  the  sample  lesson.  Each  feature  was 
referenced  to  the  criterion  within  the  baseline  criteria  on 
which  the  feature  was  based.  As  an  evaluation 
progressed,  evaluators  made  notes  for  each  feature 
regarding  the  performance  of  the  system  on  that  feature 
and  the  potential  impact  on  the  corresponding  criterion. 
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Upon  completion  of  an  evaluation,  the  evaluation 
forms  were  submitted  to  the  team  leader.  The  team 
leader  constdidated  the  comments  into  an  evaluation 
report  and,  in  consultation  with  other  team  members, 
revised  the  criteria  acc(»dingly.  Each  evaluation  report 
described  the  capabilities  of  the  system,  the  extent  to 
which  it  was  able  to  accommodate  the  sample  lesson, 
and  the  resulting  modifications  to  the  criteria. 

After  the  evaluation  team  completed  its 
assessments,  an  authoring  environment  specification 
was  fmalized,  an  RFP  drafted,  and  a  competitive 
procurement  initiated. 


SPECIFICATION  DEVELOPMENT 

The  starting  baseline  criteria  and  the  expectations 
of  the  evaluation  team  initially  reflected  an  authoring 
system  characterized  by  a  single  program  from  which  an 
author  would  create  courseware  ^nctionality.  Over  the 
course  of  the  evaluation,  however,  it  became  apparent 
that  this  concept  was  inadequate.  As  the  evaluations 
{HOgressed  and  development  requirements  came  into 
sharper  focus,  the  scope  of  the  authoring  system  (as 
{Heviously  envisioned)  broadened  to  accommodate  the 
emerging  development  requirements.  The  original 
concept  of  the  authrxing  system  was  replaced  by  a 
conception  of  a  more  dynamic  environment  with  a 
flexible  range  of  authoring  tools.  The  new  concept 
included  the  original  "authoring  system"  in  addition  to 
other  development  fuiKtions  considered  necessary  for 
highly  efficient  courseware  production.  These 
functional  elements  include  use  of  Microsoft  Windows, 
productivity  tools,  extendibiUty,  functional  graphics, 
and  rapid  prototyping. 

Windows.  From  a  productivity  standpoint,  the 
evaluation  team  decided  that  the  intercompatibility  of 
windows  applications  would  be  valuable.  The  very 
nature  of  the  authoring  environment  concept  is 
dependent  upon  the  intercompatibility  of  various 
development  elements,  such  as  graphics  files,  database 
structures,  and  courseware  executables.  This  is  the 
same  principle  on  which  the  Windows  environment  is 
based.  Thus,  the  requirement  for  the  authoring 
environment  to  be  within  the  windows  framework  is  the 
first  step  toward  the  realization  of  the  authoring 
environment  concept.  In  addition,  from  a  project 
administration  standpoint,  having  the  report  generation 
and  spreadsheet  capability  resident  within  the  authoring 
environment  has  the  potential  of  increased  productivity. 

Productivity  Tools.  The  ability  to  create  databases 
or  groups  of  instructional  templates  would  be  necessary 
to  significantly  enhance  productivity.  Since  a 
substantial  portion  of  the  intended  ICW  is  based  on 
previously  defined  instructional  strategies,  these 


strategies  can  be  represented  as  templates  (v  structures 
within  the  authoring  environment,  thus  eliminating  the 
need  to  recreate  the  structure  for  a  strategy  every  time 
the  strategy  is  employed  in  the  courseware.  While  the 
concept  of  an  instructional  template  is  not  new,  the 
system's  added  ability  to  employ  an  instructional 
template  and  actually  construct  the  course  specific  flow 
without  the  benefit  of  graphic  files,  audio  files,  video 
segments,  and  text  files  represents  a  unique  and  highly 
(B'oductive  function.  This  virtual  separation  of  content 
from  flow  enables  the  author  to  construct  an  assessable 
course  executable  prior  to  the  completion  of  other 
{nxxluction  activities.  This  would  allow  lesson  flow  and 
branching  to  be  tested  prior  to  the  insertion  of  content. 
The  ability  to  create  other  types  of  course-specific 
templates  is  included  in  this  requirement.  The 
evaluation  team  appreciated  the  developer's  need  for 
productivity  tools  specific  to  the  production  process 
and,  as  a  result,  inccnporated  the  requirement  fen-  a  built- 
in  template  capability. 

Extendibility.  The  evaluation  team  determined  that 
developers  would  require  the  ability  to  create  custom 
developed  courseware  elements  and  integrate  these 
elements  with  the  authoring  system  executables.  For 
example,  simulation  elements  frequently  require 
development  in  a  programming  l^guage  such  as  "C"  or 
assembler.  The  authoring  system  should  be  able  to  load 
and  execute  those  elements  in  a  manner  simple  for  the 
author  and  transparent  to  the  trainee.  Therefore,  the 
ability  to  utilize  such  elements  was  established  as  a 
requirement  The  extendibility  requirement  also 
encompasses  custom-develc^)^.  course-specific  utilities. 
As  an  example,  some  of  the  Navy  ICW  will  require  the 
display  and  manipulation  of  LOFAR  grams  in  Edition 
to  stand-alone  LOFAR  grams.  Thus,  a  LOFAR  gram 
generator  utility  is  needed  as  a  separate  entity  fiom  the 
authoring  system.  Since  the  LOFAR  grams  are  to  be 
an  integral  part  of  the  courseware,  the  LOFAR  gram 
utility  must  be  fully  compatible  with  the  authoring 
system  fiom  both  a  software  and  authoring  standpoint. 

It  is  this  relationship,  separate  yet  fully  compatible, 
that  defines  the  utility  as  a  part  of  the  authoring 
environment  and  mates  the  extendibility  feature  the 
cornerstone  of  the  environment  concept. 

Functional  Graphics.  Development  requirements 
clearly  indicated  that  courseware  would  have  to  simulate 
graphically  the  look  and  function  of  control  panels. 

This  involves  dials,  switches,  meters,  scopes,  and  other 
types  of  controls.  Evaluators  determined  that  these 
images,  along  with  their  corresponding  functionalities, 
should  be  reusable.  This  meant  that  in  addition  to 
storing  the  image  of  a  particular  type  of  dial,  its 
corresponding  movements  should  te  stored  along  with 
it.  This  would  enable  authors  to  construct  functional 
control  panels  by  linking  previously  created 
"functional”  graphics  using  the  authoring  system. 
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Rapid  Prototyping.  It  is  often  the  case  that 
developers  learn  development  lessons  too  late  in  the 
production  cycle.  The  ability  to  rapid  prototype,  or 
quickly  produce  pcations  of  lessons,  can  provide 
valuable  information  regarding  the  feasibility  of  certain 
instructional  cn*  developmental  approaches.  This 
information  can  be  used  to  avoid  costly  and  time 
consuming  re-development  The  evaluators  therefore 
decitted  to  require  a  rapid  prototype  capability  that  would 
employ  the  previously  described  productivity  tools. 

In  addition  to  these  functional  elements  that  help 
to  define  the  concept  of  an  authoring  environment,  there 
are  several  other  features  of  the  actual  authoring  process 
that  the  evaluators  determined  merited  special  attention 
in  the  functional  specification.  Testing  and  data 
collection  features  are  described  in  detail  because  of 
numerous  weaknesses  discovered  in  this  area  during  the 
evaluations.  The  ability  to  synchronize  digital  audio, 
video,  and  graphics  became  another  requirement,  as  a 
result  of  problems  encountered  during  the  evaluations. 
The  evaluators  also  identified  symbolic  representation  of 
course  structure  as  a  requirement.  It  was  determined  that 
the  ability  to  depict  courseware  flow  and  structure,  as  in 
a  block  flow  diagram,  was  an  essential  development  aid. 
Table  1  illustrates  all  of  the  categories  and  sulxategories 
in  the  final  functional  specification  formulated  by  the 
evaluation  team. 


CONCLUSION 

The  evolution  from  the  initial  oaaeune  criteria  to 
the  final  specification  reflected  some  basic  changes  in 
philosophy  regarding  the  authoring  process.  Traditional 
authoring  systems  of  the  last  decade  were  star  J-alone 
tools  allowing  designers  and  -’•’th-.s  to  create  simple, 
tutorial  type  presentations.  Ine  design  of  the 
courseware  was  inhibited  by  the  limitations  of  the 
authoring  system.  With  the  requirements  of  large  scale, 
high  level  instruction  delivered  on  an  interactive  system 
simulating  a  complex  device,  the  authoring  component 
of  production  has  been  redefined.  As  a  result,  an 
authoring  environment  consisting  of  flexible 
p-oductivity  tools  and  functions  in  addition  to  the 
traditional  authoring  system  was  conceived.  This 
"authoring  environment"  represents  the  next  step  in 
interactive  training  p’oduction. 
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Table  1.  Authoring  Environment  Specifications 


FEATURES 

DISPLAY 

GRAPHICS 

AUDIO/ 

VIDEO 

INPUT 

PROCESSING 

TESTING 

INTERFACE 

Dual  Mooitors 

Internal  Graphics 

Video  Editor 

Penpheral  Input 

Variables 

Tests 

User  Interface 

-  Graphic  FutKtions 

-  Video  Stills 

-  Test  Data 

-  Editor 

Video  Playback 

•  Funcliorul  Graphics 

•  Audio  Channels 

Senal  IA3 

Braacbing 

•  Tesi  Banks 

•  Test  Integrity 

Productivity  Tools 

Text  Imponiog 

External  Graphics 

Graphic  Overlay 

Touch  Areas 

External  Executables 

•  Rapid  Prototyping 

-  Graphic  Overlays 

Procedural  Testmg 

-  Extendiblity 

Digital  Audio 

Questions 

Memory  Resident 

-  Response  Latency 

•  Response 

Utilities 

Pilot  Testing 

Syochromzacion 

Submission 
•  Answer  Judging 

Data  Coliectioo 

Questions 

Help 

•  Data  Files 

Input  Devices 

Multiple  Inputs 

Objective  Tracking 

Computer  Generated 
Sounds 

Error  Checking 
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ABSTRACT 


A  comparative  analysis  of  current  media  selection  models  was  conducted  as  part 
of  an  overall  effort  to  improve  key  Instructional  Systems  Development  (ISD) 
steps  that  would  significantly  benefit  from  automation  and  decision  support 
features.  The  media  selection  step  within  the  ISD  process  was  chosen  for  in- 
depth  evaluation  of  its  automation  requirements.  The  objectives  of  this  study 
were  threefold:  (a)  to  compare  and  contrast  the  media  selection  models  in 
general  use  by  military,  contractor,  and  civilian  instructional  designers;  (b) 
to  identify  key  features  that  impact  media  selection  model  automation;  and  (c) 
to  develop  a  preliminary  design  of  an  automated  media  selection  model  that 
incorporates  the  requirements  of  as  many  of  the  reviewed  media  selection 
models  as  possible. 


DR  H.  BARBARA  SORENSEN,  a  senior  research  scientist  with  the  US  Air  Force 
Armstrong  Laboratory,  possesses  a  Master’s  degree  in  Educational  Psychology, 
Measurement,  and  Statistics  and  a  Doctorate  in  the  areas  of  instructional 
Design  and  Computer  Technology,  Her  experience  spans  15  years  of  government 
involvement  in  research  and  development  and  military  training  for  the  Army, 
Navy,  and  Air  Force.  She  also  has  several  years  of  experimental  research  and 
development  experience  with  university  systems,  developing  educational 
programs  for  elementary,  secondary,  and  college  institutions.  Currently,  she 
manages  Joint  Service  advanced  programs,  integrating  the  logistic  and  training 
processes  in  the  mixitary  environment,  in  addition  to  managing  several 
programs  to  model  the  integration  of  manpower,  personnel,  and  training 
factors . 
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AN  ASSESSMENT  OF  TRAINING  &  EDUCATION  MEDIA  SELECTION  MODELS 


INTRODUCTION 

The  media  selection  analysis  step  of 
the  Instructional  Systems  Development 
(ISD)  process  has  been  the  focus  of 
continual  research  by  instructional 
designers  and  training  developers. 

This  research  has  led  to  countless 
attempts  to  develop  workable  models  to 
manage  the  ISD  process  (Anderson,  83; 
Braby,  73;  Kemp  &  Dayton,  85; 

Merideth,  65;  Reiser  &  Gagne,  82). 

The  ISD  literature  is  profuse  with 
media  selection  models  and  each  new 
model  developer  approaches  the  subject 
with  the  belief  that  this  attempt  will 
be  better  than  all  previous  models. 
Prior  to  developing  a  media  model,  the 
literature  is  reviewed,  the 
shortcomings  are  pointed  out,  and  a 
new  model  is  created.  Determining 
which  model  is  the  bestbecomes  the 
subject  of  much  debate  within  the  ISD 
commun ' ty . 

The  purpose  of  this  effort  was  not  to 
create  another  media  selection  model. 
The  intent  was  to  identify  and 
evaluate  the  current  models  and  to 
develop  a  flexible,  automated 
environment  in  which  the  leading  media 
selection  models  could  be  hosted. 

Much  of  the  justification  for  this 
approach  relies  from  past  media 
selection  research. 

Several  excellent  reviews  of  the  media 
selection  research  include  Lumsdaine 
(1963),  Lumsdaine  and  May  (1965), 
Campeau  (1967)  in  Briggs,  (iampeau, 
Gagne,  and  May  (1967),  Allen  (1971), 
Saettler  (1968),  Meredeth  (1965),  and 
Tosti  and  Ball  (1969).  All  of  these 
reviews  referred  to  the  hundreds  of 
studies  that  generally  concluded  with 
a  finding  of  "no  significant 
difference."  These  results  were 
obtained  because  of  a  failure  to 
control  the  complex  interplay  of 
instructional  variables  such  as  media, 
learner,  subject  matter,  and  course. 
These  variables  continually  have  been 
confounded  in  most  of  the  media 
studies  (Sorensen,  Park,  &  Awtry, 

1990) .  Nearly  all  educational  and 
training  media  researchers  concluded 
that  basic  well-designed  media 
res'^arch  has  yet  to  be  performed  that 
woula  prove  the  efficacy  of  any  medium 
of  instruction  over  another.  Of  more 


fundamental  concern  was  the  general 
lack  of  agreed-upon  media  taxonomies 
and  instructional  design  parameters  in 
which  to  conduct  that  research. 
Accordingly,  most  mr.ia  selection 
models  were  generall'  based  upon 
limited  research  fir  dings  and 
substantial  conjecture. 

Until  this  fundamental  research  is 
performed,  the  best  media  selection 
environment  that  can  be  devised  is  one 
that  is  highly  flexible  in  the  manner 
that:  (1)  the  media  attributes  (e.g., 
display  characteristics,  stimulus 
medium,  level  of  behavior)  and  media 
(e.g.,  videotape,  still/visual,  part- 
task  trainer)  are  defined  and 
ssociated  to  each  other;  (2)  the 
model’s  design  is  presented  and 
executed  by  the  user;  and  (3)  the 
results  are  aggregated  and  presented 
for  evaluation.  If  changes  dictated 
by  new  research  findings  and  new 
technological  changes  in  media  are  to 
be  incorporated  into  a  model’s  logic, 
user-control  of  the  media  selection 
model  is  of  prime  importance. 


Media  Selection  Models  Analysis 

The  analysis  described  here  identified 
and  evaluated  over  50  media  s. 'lection 
models.  Initially,  as  many  different 
models  as  possible  were  identified  and 
obtained  for  review.  Candidate  models 
represented  a  broad  cross  section  of 
military,  contractor,  and  civilian 
users,  different  levels  of  detail  and 
comlexity,  different  end  uses  (e.g., 
front-end  analysis  versus  curricula 
development,  and  different  decision¬ 
making  approaches  e.g.,  matrix, 
flowchart,  etc).  Of  special  interest 
were  models  that  had  been  automated. 
The  models  that  are  known  to  have  been 
automated  include  3306th  Training 
Development  and  Evaluation  Squadron 
(TDES),  Automated  Instructional  Media 
(AIMS)  Selection,  Computer-Aided 
System  for  Developing  Aircrew  Training 
(CASDET),  Melton,  Man  Integrated 
Systems  Technology  (MIST) ,  Training 
Analysis  Support  System  (TASCS), 
Technique  for  Choosing  Cost  Effective 
Instructional  Procedures  (TECEP), 
Training  Efficiency  Estimation  Model 
(TEEM),  and  Training  Technology 
Application-Decision  Support  System 
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(TTA-DSS) . 

Many  of  the  models  had  been  identified 
in  previous  reviews  of  media  selection 
models .  Most  developers  of  new  models 
included  a  review  of  previous  models, 
using  identified  shortcomings  as  the 
basis  for  formulating  new  models. 
Notable  reviews  included  those  by 
Braby  (1973),  Briggs  (1967),  Main  and 
Paulson  (1988),  Melton  (1988),  Reiser 
and  Gagne  983),  Romiszowski  (1988), 
and  Spangenberg,  Riback  and  Moon 
(1973) . 

After  initial  evaluation  and 
assessment,  the  list  of  50  media 
models  was  reduced  to  32.  These  32 
models  selected  for  inclusion  in  this 
report  are  stimmarized  in  Table  1. 

Many  of  the  50  models  initially 
identified  were  deleted  because  of 
inadequate  or  dif f icult-to-obtain 
documentation.  Most  of  the  deleted 
models  are  obscure  and  the  difficulty 
in  obtaining  their  documentation  may 
be  an  indication  that  they  are  no 
longer  available  or  used.  Some  of  the 
deleted  models  had  very  poor 
documentation  or  just  did  not  meet 
minimal  criteria  to  be  called  a  media 
selection  model.  Unfortunately,  a  few 
of  the  more  desirable  models  were 
developed  by  contractors  and  were 
unattainable  because  of  their 
proprietary  nature.  Altogether,  the 
32  surviving  models  represent  a 
comparable  cross  section  of  the  models 
in  use  today.  The  names  associated 
with  the  models  in  Table  1  are  those 
that  are  generally  used  to  refer  to 
them  by  instructional  designers. 

The  second  column  in  Table  1  indicates 
whether  the  model  was  intended  to  be 
applied  within  the  context  of  training 
front-end  analysis  (FEA)  or  as  part  of 
traditional  instructional  systems 
development  (ISD).  Within  the 
military,  FEA  requirements  are 
typically  met  through  contractors 
applying  Military  Standards  and  their 
related  Data  Item  Descriptions  (DlDs). 
ISD  requirements  are  often  achieved  by 
government  personnel  using  one  of  the 
many  ISD  manuals  and  pamphlets 
(Sorensen,  1990).  As  shown  in  Table 
1 ,  only  8  of  the  models  are  intended 
for  FEA  use  as  a  large  number  of 
civilian  school  models  on  the  list  are 
not  intended  for  this  purpose.  Also, 
the  unavailable  contractor  models 
account  for  some  of  these  low  numbers. 


The  third  column  of  Table  1 
indicates  the  type  of  behavioral 
statement  that  the  model  is  Intended 
to  analyze.  The  term  behavioral 
statement  is  used  throughout  this 
report  to  refer  to  tasks,  task 
elements,  skills  and  knowledge,  and 
learning  objectives.  Nearly  all  of 
the  models  required  learning 
objectives  as  their  input.  Only  5  of 
the  models  were  designed  for  use  with 
tasks.  Not  surprisingly,  all  but  one 
of  these  was  intended  to  be  used  as 
part  of  FEA.  Only  the  3306th  TDES  and 
the  TTA-DSS  models  performed  the  media 
selection  process  at  the  skills  and 
knowledge  level  (Sorensen,  1987).  The 
3306th  TDES  model  performed  its 
analysis  wholly  at  this  level  of 
detail.  The  TTA-DSS  provided  this 
capability  as  an  option.  Some 
of  the  models  permitted  analysis  of 
both  tasks  and  learning  objectives. 

All  of  the  32  models  are  designed  for 
one  of  six  basic  forms  in  the 
decision-making  process;  (1) 
guideline,  (2)  worksheet,  (3)  table, 
(4)  matrix,  (5)  matrix  with  weights, 
or  (6)  flowchart.  These  forms 
represent  the  interface  that  leads  the 
analyst  through  the  model’s 
decision-making  logic.  The  type  of 
decision  making  form  is  indicated  in 
column  six  of  Table  1  and  a  detailed 
discussion  of  the  forms  follows. 

Almost  all  of  the  models  incorporated 
a  single  version  of  the  decision 
making  form,  e.g.,  the  AIMS  model  used 
a  single  matrix  with  weights. 

However,  a  few  of  the  models  employed 
multiple  versions  of  a  decision  making 
form.  In  these  "multiple  form" 
models,  the  behavioral  statements  were 
initially  categorized  and  a  different 
version  of  a  decision  making  form  was 
applied  to  the  category.  Two 
approaches  to  categorization  were 
found  in  the  models:  one  approach 
assigned  the  behavioral  statements  to 
a  type  of  learning  category  (e.g., 
cognitive,  affective,  psychomotor)  and 
the  other  approach  made  the  assignment 
to  a  phase  of  learning  category  (e.g., 
actuate  motivation,  transfer  of 
learning,  provide  feedback).  The 
model  requirements  for  these 
approaches  are  shown  as  columns  four 
and  five  in  Table  1.  These  approaches 
are  discussed  later  followed  by  a 
review  of  the  attributes  and  questions 
that  were  used  in  the  media  selection 
models.  Similarly,  an  assessment  of 
the  media  outcomes  that  were  obtained 
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from  the  models  will  be  discussed  as 
well  as  the  use  of  media  selection 
models  within  the  context  of  a  course 
of  instruction. 

Decision-Making  Process 

Other  media  selection  model  reviews 
(Gagne,  1982  and  1983;  Main  and 
Paulson,  1988;  Melton,  1988;  and 
Romiszowski,  1988)  have  identified  a 
number  of  different  decision  making 
forms  or  formats.  This  review  differs 
from  the  others  in  terms  of  the  number 
of  models  reviewed  and  the  purposes 
for  which  the  reviews  were  performed. 
In  this  review,  format  categories  have 
been  identified  that  best  characterize 
model  types  from  an  automation 
standpoint . 

Across  all  models,  seven  different 
types  of  decision  making  forms  were 
identified:  (1)  guideline,  (2) 
worksheet,  (3)  table,  (4)  matrix,  (5) 
matrix  with  weights,  (6)  flowchart, 
and  (7)  multiple  form.  Table  2 
summarizes  the  media  selection  models 
by  decision  making  form.  Detailed 
descriptions  of  each  of  these  forms 
follow. 

The  guideline  decision  making  form  is 
the  most  unstructured  and  the  most 
difficult  to  format  for  automation. 
Examples  of  these  models  include 
Cavert  (1972),  Goodman  (1971), 

Heinich,  Molenda,  and  Russell  (1989), 
Levie  (1975),  an  Tosti  and  Ball 
(1969).  Primarily,  these  models,  as 
well  as  a  number  of  others  that  were 
rejected  for  inclusion  in  this  study, 
were  initial  attempts  at  developing 
selection  models.  As  Romiszowski 
(1988)  has  pointed  out,  the  1970s  was 
the  decade  for  media  selection.  Some 
models  in  this  form  (Tosti  and  Ball, 
1969)  are  probably  best  classified  as 
research  products  and  not  fully 
developed  models  Intended  for 
application.  These  models  are  mostly 
narrative  in  nature  and  provide 
little,  if  any,  firm  direction  or  aids 
to  perform  the  selection  process. 

This  decision  making  form  was  rejected 
from  consideration  for  automation. 

The  worksheet  decision  making  form  is 
more  structured;  however,  each 
worksheet  is  unique  to  the  media 
selection  model.  Models  that  employed 
this  approach  include  AFP  50-58 
(1978),  Briggs  and  Wager  1981),  and 
Gerlach  and  Ely  (1971). 


The  AFP  50-58  (1978)  model  is  a 
complex  worksheet .  After  classifying 
the  learning  objective  into  categories 
of  concrete  and  abstract,  the 
instructional  analyst  must  then 
determine  media  for  each  of  three 
stages  of  learning;  early,  middle,  and 
late.  Different  presentation  mode 
decisions  are  then  made  for  each 
category.  Presentation  mode  entries 
on  the  worksheet  do  not  lead  to  a 
recommended  set  of  media.  Rather,  the 
instructional  analyst  has  to  mentally, 
and  with  the  aid  of  a  media  directory, 
combine  the  pattern  of  presentation 
mode  choices  into  a  best  medium.  If 
this  pattern  of  selections  were  used 
by  the  model  to  match  up  to  patterns 
in  a  pre-set  media  pool,  then  this 
model  would  represent  a  matrix 
decision  making  form.  The  Briggs  and 
Wager  (1981)  model  is  a  true 
worksheet.  This  format  is  unique  to 
this  model  and  is  set  up  to  record  the 
analyst’s  decisions  in  a  narrative 
manner.  The  worksheet  decision  making 
form  is  considered  to  be  too  variable 
and  unique  across  models  to  be 
consistently  automated.  Moreover, 
this  approach  is  generally  too 
open-ended  and  subjective  in  its 
application. 

A  table  decision  making  form  is 
similar  to  the  following  matrix  form. 

A  table  has  media  attributes  and  media 
arranged  in  the  rows  and  columns  of  a 
matrix.  However,  the  values  obtained 
at  the  intersection  are  not  clear  and 
are  typically  qualified  with  notes  and 
side  comments  about  the  goodness  of 
fit.  This  form  could  be  automated  if 
more  precise  and  consistent  values 
were  used. 

Some  of  these  table  models  can  be 
easily  converted  to  a  matrix  by 
assigning  numeric  values  to  the 
decision  points.  For  example,  models 
that  use  high,  medium,  and  low  at  the 
decision  points  can  be  converted  to 
values  of  1,  2,  and  3.  In  this 
manner,  the  model  could  be  used  as  a 
matrix  with  weights  form.  Models  that 
belong  to  this  decision  making  form 
include  Allen  (1967),  Boucher, 
Gottlieb,  and  Morganlander  (1973) 
MCDECO  P1553.1A,  and  Wilshusen  and 
Stone  (1970). 

The  matrix  decision  making  form  is  the 
first  form  tyie  that  can  be  readily 
automated.  The  AV-8B  Model  is  an 
example  of  a  matrix  form.  Models  in 
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this  category  have  rows  and  columns 
that  Include  media  attributes  or 
characteristics  and  media  categories 
or  media.  At  their  intersection,  an 
"X"  or  yes  indicates  a  match  between 
an  attribute  and  a  medium.  A  match 
can  then  be  made  between  a  behavioral 
statement’s  attribute  set  and  the 
media  pool’s  attribute  assignments  to 
find  the  best  medium.  The  media  can 
be  rank-ordered  by  the  number  of 
attribute  matches  to  identify 
alternative  media.  Models  that  use 
this  form  include  AV-8B  (1981),  CASDAT 
(1983),  McConnell  (197A),  Melton 
(1988),  MIST  (1985),  and  TEEM  (1981). 

The  matrix  with  weights  decision 
making  form  is  similar  to  the  previous 
form,  except  a  numerical  value  is 
entered  into  the  matrix.  This  form 
type  can  also  be  automated.  The  AIMS 
model  uses  this  approach  in  that 
numerical  values  correspond  to  a 
predetermine  Likert  scale  that  ranges 
from  0  to  5.  Zero  indicates  no  match 
between  the  attribute  and  the  medium. 
The  values  from  1  to  5  indicates  some 
degree  of  match.  A  "1"  indicates  the 
lowest  match  and  a  "5"  indicates  the 
highest  match.  The  scale  points  have 
different  names  to  indicate  different 
attribute  metrics  to  be  measured. 

Media  choices  are  made  by  averaging 
the  attribute  values  and  rank-ordering 
the  results.  In  addition  to  AIMS 
(1983),  this  form  type  includes  TASCS 
(1987)  . 

The  flowchart  or  decision  tree  is 
another  decision  making  form. 

Examples  of  this  form  include  3306th 
TDES  (1987),  Anderson  (1983),  Bretz 
(1971a),  Kemp  and  Dayton  (1985), 

Reiser  and  Gagne  (1983),  and 
Romiszowski  (1988).  In  this  form 
type,  the  user  is  led  through  the 
decision-making  process  via  a  series 
of  questions.  The  questions  can  be 
either  linear  or  branching.  In  all  of 
the  flowchart  models  found,  the  final 
decision  at  the  end  of  the  questions 
was  a  media  class,  not  a  single,  best 
medium.  Each  media  class  or  category 
included  a  number  of  media  examples. 
From  an  automation  standpoint,  the 
flowchart  can  be  converted  to  a 
matrix.  One  way  to  correct  the 
flowchart  method  is  to  change  yes /no 
questions  to  "Is"  and  "Os"  in  the 
matrix  as  was  done  by  the  Reiser  and 
Gagne  model.  The  3306th  TDES  model 
consists  of  seven  linear  questions  to 
make  hardware  decisions,  followed  by 


five  branching  questions  to  determine 
media,  and  is  connected  to  a  matrix 
format . 

Three  of  the  models  evaluated  (AFM 
50-2,  1970;  F-16,  1981;  and  TECEP, 
1978)  required  more  than  one  decision 
making  form.  All  of  the  forms 
required  by  a  model  were  of  the  same 
type.  There  are  two  "matrix  with 
weights"  forms  required  by  the  F-16 
model.  In  this  model,  learning 
objectives  are  first  classified  into 
hands-on  (training  devices)  and 
academic  categories .  A  unique  matrix 
with  weights  is  then  used  for  each 
category  of  learning  objectives. 

One  of  the  matrices  used  in  the  TECEP 
(1978)  classifies  the  learning 
objectives  into  12  types  of  learning 
and  a  different  matrix  algorithm  is 
applied  to  each  type.  Both  of  these 
models  require  an  initial  learning 
classification  capability  to 
categorize  each  learning  objective.  A 
description  of  a  learning 
classification  capability  that  will 
meet  both  model  needs  was  developed. 

In  addition  to  the  learning 
classification  capability,  the  AFM 
50-2  (1979)  model  requires  a 
capability  to  assign  learning 
objectives  to  learning  phases.  This 
model  requires  learning  objectives 
first  be  assigned  to  one  of  six  types 
of  learning.  Learning  objectives  are 
then  assigned  to  the  learning  phases 
of  presentation,  practice,  and 
feedback.  It  must  be  noted  in 
evaluating  the  AFM  50-2  model,  that 
using  the  previous  decision  making 
form  description,  this  model  is  really 
a  table.  It  is  assumed  that  the 
values  (Yes,  P,  No,  and  N/A)  can  be 
converted  to  numerical  values  (3,  2, 

1 ,  and  0 ) . 

Learning  Classification  Schemes 

Table  3  provides  a  summary  of  the 
learning  classifications  used  by  some 
of  the  media  selection  models.  The 
type  of  learning  classification  is  an 
important  factor  in  many  of  the  models 
as  shown  in  that  Table.  Learning 
classification  is  used  here  to 
represent  the  type  of  learning  outcome 
that  the  behavioral  statement 
represents.  The  number  of 
classifications  vary  from  the  three 
high-level  categories  of  cognitive, 
psychomotor,  and  affective  as  used  by 
AIMS  (1983),  Anderson  (1983),  and 
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Durham  (197A)  to  the  very  detailed, 
12-category  TECEP  model.  Some  of  the 
learning  classification  schemes  are 
well-known  and  established.  The  AFM 
50-2  (1979)  and  AFP  50-58  (1978) 
models  are  based  in  part  on  the 
learning  classification  scheme  of 
Gagne;  the  Melton  (1988)  model  is 
based  on  Bloom  (1956);  and  the  TECEP 
(1978)  model  is  based  on  Aagard 
(1976).  Interestingly,  the  Reiser  and 
Gagne  (1983)  model  does  not  use  the 
complete  Gagne  (1985)  scheme  of 
learning  categories. 

For  most  of  the  models,  learning 
classification  is  used  as  a  selection 
attribute  or  question  in  the  model. 
However,  for  some  models,  the  learning 
categories  created  by  the 
classification  scheme  become  the  basis 
for  employing  a  different  media 
selection  algorithm.  These  models  are 
considered  as  "multiple  form."  As 
previously  indicated,  the  TECEP  (1978) 
model  requires  the  classification  of 
learning  objectives  into  12  types  of 
learning.  A  different  media  selection 
matrix  is  then  applied  to  each  type  of 
learning.  A  similar  classification 
approach  is  used  by  the  AFM  50-2 
(1979)  model.  In  this  multiple  form 
model,  learning  objectives  are 
classified  into  6  elements  of  learning 
strategies . 

The  F-16  (1981)  model  represents  a 
special  case  of  learning 
classification.  This  multiple  form 
model,  as  well  as  the  AV-8B  (1981) 
model,  was  developed  in  response  to  a 
front-end  analysis  requirement  to 
perform  media  selection  in  accordance 
with  military  specification 
MIL-T-29053B .  The  media  selection 
data  item  descriptions  (DI-H-25717  and 
DI-H-25720)  require  a  model  that 
separately  evaluates  hands-on  and 
academic  media  requirements.  The  F-16 
(1981)  model  requires  the  learning 
objectives  to  be  separated  into  two 
categories:  academic 
and  training  device.  Two  different 
matrices  are  then  applied  to  the  two 
learning  objective  sets.  In  the  AV-8B 
(1981)  model,  the  developer  uses  the 
same  matrix  to  perform  both  analyses. 

For  these  three  multiple  form  media 
selection  models,  an  automated 
learning  classification  capability 
similar  to  that  used  in  the  Learning 
Objectives  Classification  Tool  (1989) 
would  greatly  assist  the 


classification  process.  Keywords 
could  be  used  in  the  same  manner  to 
assign  the  behavioral  statements  to 
the  appropriate  categories. 

Learning  Phases 

Another  critical  design  element  of 
media  selection  models  is  the 
assignment  of  learning  phases  to 
learning  objectives.  Table  A  shows 
the  five  models  that  employ  this 
approach.  The  phrase  "phases  of 
learning"  is  used  broadly  here  to 
indicate  the  stages  or  events  of 
instruction.  As  with  learning 
classification  schemes,  phases  of 
learning  are  used  in  some  models  as 
attributes  and  in  at  least  one  model 
as  separate  categories  that  require 
application  of  different  media 
selection  algorithms. 

In  addition  to  the  learning 
classification  requirement  described 
previously,  the  AFM  50-2  (1979)  model 
further  classifies  each  learning 
objective  by  the  learning  strategy 
elements  of  presentation,  practice, 
and  feedback.  For  each  learning 
classification  and  learning  phase  set 
a  different  media  selection  matrix  is 
applied.  The  AFM  50-2  model  is  the 
only  model  found  that  results  in  a 
multiple  form  designation.  No 
automated  job  aid  to  help  perform  this 
assignment  seems  possible. 

Media  Categories 

Table  5  shows  the  media  incorporated 
in  a  sampling  of  models .  Some  of  the 
models  cluster  media  by  categories. 

The  categories  represent  generic  or 
psychological  media  capabilities 
(e.g.,  still  visual,  tutorial  media, 
sound/moving  visual,  etc).  Within 
these  categories  are  found  the  actual 
media  channel  or  hardware  from  which 
to  choose  (e.g.,  slides,  CAI/CMI, 
television,  etc).  The  MIST  (1985) 
model  is  the  only  model  that  uses 
generic  media  throughout  (e.g., 
audiovisual  motion  with  feedback, 
etc) . 

Course  Requirements  and  Media 
Selection 

Few  models  uses  course  requirements  as 
an  analysis  component  of  the  media 
selection  process.  When  mentioned  as 
a  criterion,  this  media  selection 
dimension  is  typically  addressed  in 
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Table  5.  Sunary  of  Selected  Media  Selection  Models  by  Media  (Concluded) 
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broad  terms.  Only  TASCS  (1987)  and 
CASDAT  (1983)  evaluate  the  media 
choices  within  some  context  of  course 
usage.  The  amount  and  ease  of  media 
use  within  the  intended  course  can  be 
an  important  selection  criteria.  This 
is  an  especially  important  step  if  a 
cost/benefit  type  of  analysis  is  to  be 
performed  on  competing  or  alternative 
media. 

Media  Selection  Model  Design 

Figure  1  represents  a  preliminary 
design  of  the  automated  media 
selection  model.  As  shown,  there  are 
four  major  functions  that  the  design 
would  support.  The  first  function  is 
a  media  selection  model  decision  aid. 

A  user  of  the  proposed  system  may 
choose  a  media  selection  model  already 
resident  within  the  system  or  the 
model  can  be  entered  using  the 
system’s  development  function.  If  the 
user  does  not  know  which  model  to  use, 
this  decision  aid  can  be  used  to 
assist  in  selecting  the  best  model. 

The  media  selection  model  decision  aid 
is  a  logic  tree  which  leads  the  user 
through  a  series  of  questions.  At  the 
conclusion,  the  user  will  be  presented 
with  one  or  more  models  that  fit  the 
user’s  analysis  parameters. 

The  second  function  provides  for  the 
development  of  a  System  Data  Set. 

This  development  is  a  three-step 
process:  (1)  a  new  System  Data  Set  is 
defined  with  a  name  and  description; 

(2)  a  media  model  is  selected  for  the 
System  Data  Set;  and  (3)  the  System 
Data  Set  is  populated  with  Behavioral 
Statements,  either  by  sorting  from 
existing  Behavioral  Statements  not 
already  included  in  a  System  Data  Set 
or  by  manually  entering  new  Behavioral 
Statements . 

The  development  of  a  media  selection 
model  is  the  third  function  and 
consists  of  the  following  steps;  (1) 
defining  a  new  model  with  a  name  and 
description;  (2)  identifying  the 
structure  (single-form  or  multi-form) 
and  the  types  of  decision  making  forms 
to  be  used;  (3)  selecting  the  actual 
forms  to  be  used,  eith.r  generic  blank 
forms  or  currently  exi  ting  forms  in 
other  models;  (4)  modifying  the  forms, 
if  needed;  blank  forms  must  be 
populated  with  appropriate  attributes, 
attribute  categories,  media,  media 
categories,  questions,  question 
categories,  weights,  question-media 


weights,  and  flow  rules;  (3)  selecting 
the  type(s)  of  Behavioral  Statements 
that  the  model  will  analyze;  (6) 
determining  what  type  of  Learning 
Classification  Scheme  is  appropriate, 
if  any;  and  (7)  building  the  Learning 
Classification  Scheme,  either  from 
scratch  or  by  copying  and  modifying 
ones  from  other  models. 

Media  selection  model  application  is 
the  fourth  function  and  has  eight 
analysis  steps  which  are  described 
below: 

(1)  Select  Media  Selection  Model. 
The  instructional  analyst  selects  (or 
creates)  a  media  selection  model  for 
the  System  Data  Set.  The  models  that 
are  available  within  the  system  are 
described  in  Table  6.  The  analyst  may 
use  any  of  these  models  or  may  enter 
another  model  using  the  previous  model 
development  function. 

(2)  Assign  Behavioral  Statements 
to  Form  Data  Set.  If  the  model 
assigned  to  the  System  Data  Set  is  a 
multiple  form  structure,  a  Form  Data 
Set  is  created  for  each  decision 
making  form.  If  the  model  is 
single-form,  then  there  is  only  one 
Form  Data  Set  created.  Each  Form  Data 
Set  is  named  and  defined  by  the 
analyst.  The  Media  Model’s  Learning 
Classification  Scheme  is  applied  to 
the  System  Data  Set’s  Behavioral 
Statements  to  assign  Types  of 
Learning.  At  the  same  time,  if  Phases 
of  Learning  are  required,  they  are 
manually  assigned.  If  the  model  is 
multiple  form,  this  serves  to  assign 
each  Behavioral  Statement  to  one  of 
the  Form  Data  Sets. 

(3)  Evaluate  Behavioral 
Statements  Using  Model.  For  each 
Behavioral  Statement,  the  appropriate 
decision  making  form  is  applied  and 
attributes  or  questions  are  selected. 
In  the  matrix  type  of  forms,  the 
analyst  manually  compares  each 
attribute  to  the  Behavioral  statement 
and  marks  it  as  yes  or  no.  In  the 
flowchart  form,  each  question  is 
answered  yes /no  in  a  branching  manner 
that  ultimately  leads  to  completion. 

(4)  Determine  Initial  Media.  The 
matrix  type  of  decision  making  forms 
apply  the  attributes  selected  against 
the  weights  for  each  Behavioral 
Statement  and  calculate  a  value  for 
the  media.  The  media  are  given  a  rank 
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Figure  I.  Preliminary  Media  Selection  Model  Design 


ry  of  Selected  Media  Selection  Models  of  Merit 


order,  and  the  first  10  media  became 
the  initial  media  set.  In  the 
flowchart  form,  the  pattern  of  yes /no 
answers  to  the  questions  points  to  a 
particular  group  of  media,  which 
becomes  the  initial  media  set. 

(5)  Determine  Training  System 
Alternatives.  The  analyst  may  choose 
to  have  several  Training  System 
Alternatives  for  the  System  Data  Set. 
These  can  also  be  thought  of  as  final 
media  versions;  for  each  Behavioral 
Statement,  a  final  medium  is  selected 
for  each  version.  Each  version  has  a 
name  and  description  and  can  be 
independently  modified  to  optimize 
media  utilization. 

(6)  Select  Final  Media.  For  each 
Behavioral  Statement,  the  analyst 
manually  selects  a  final  medium  to  go 
with  each  Training  System  Alternative. 
These  may  be  selected  from  the 
Behavioral  Statement’s  initial  media 
set  or  else  from  the  general  media 
database . 

(7)  Cluster  Media  Results.  From 
time  to  time  during  the  media 
selection  process,  either  initial 
media  or  final  media  may  be  sorted  in 
a  wide  variety  of  ways.  The  sort 
statements  are  called  clusters  and  may 
be  stored.  The  results  are  used  in  ad 
hoc  reports  to  aid  the  analyst  in 
making  media  assignment  decisions. 

(8)  Determine  Course  Media 
Requirements.  For  each  Training 
System  Alternative,  the  course  has  a 
separate  schedule  and  every  lesson  in 
it  has  a  separate  lesson  media  set. 

The  analyst  develops  lesson  media  sets 
by  examining  the  final  media  of  all 
the  Behavioral  Statements  in  the 
lesson  and  manually  selecting  one  or 
more  for  lesson  media  set.  Clustering 
and  ad  hoc  reports  may  play  a  role  in 
the  analyst’s  decision-making.  A 
course  schedule  may  also  be  modified 
to  optimize  media  utilization. 

CONCLUSIONS 

In  conducting  this  study,  over  50 
media  selection  models  were  identified 
and  evaluated.  Thirty-two  of  these 
models  were  further  evaluated  and 
documented.  The  models  represent  a 
cross-section  of  military,  contractor, 
and  civilian  education  approaches  to 
media  selection.  Nearly  all  of  the 
models  are  designed  to  evaluate  tasks 


and  learning  objectives.  Twenty-eight 
of  the  32  models  are  intended  to  be 
used  with  learning  objectives. 

Seven  different  decision  making  forms 
were  identified:  (1)  guideline,  (2) 
worksheet,  (3)  table,  (A)  matrix,  (5) 
matrix  with  weights,  (6)  flowchart, 
and  (7)  multiple  form.  The  first 
three  forms  are  too  unique  in 
application  to  be  readily  automated. 
Each  of  these  forms  is  tailored  to  the 
model’s  requirements  and  utilizes 
formats  that  vary  widely  across 
different  models.  Moreover,  models 
that  use  these  forms  tend  to  provide  a 
loose  analysis  audit  trail  and  lead 
ultimately  to  more  subjective  results. 
These  approaches  do  not  provide 
sufficient  analysis  justification  of 
results  required  by  most  military 
training  decision  makers. 

The  table  decision  making  form  is 
similar  to  the  matrix  form. 

Converting  the  tables  decision  values 
to  yes /no  responses  or  to  a  numeric 
weight  permits  their  inclusion  in  the 
proposed  design. 

The  remaining  four  decision  making 
forms  can  be  hosted  by  the  proposed 
design.  These  decision  forms 
represent  17  of  the  32  models 
evaluated.  Most  of  the  military 
models  are  represented  in  this  group. 

A  single  version  of  a  decision  making 
form  is  used  in  29  of  the  32  models . 
The  remaining  3  models  (AFM  50-2, 

1979;  F-16,  1981;  and  TECEP,  1978)  use 
a  multiple  form.  These  models  use 
more  than  one  version  of  the  same 
form.  Assigimient  to  the  different 
forms  is  performed  using  one  of  two 
approaches . 

The  above  three  models  use  a  learning 
classification  scheme  to  create 
subsets  of  behavioral  statements 
(i.e.,  tasks,  learning  objectives, 
etc.).  This  assignment  of  all 
behavioral  statements  into  separate 
form-unique  categories  precedes 
application  of  each  form’s  decision 
making  logic .  It  is  recommended  that 
an  automated  learning  classification 
capability  be  embedded  in  the  proposed 
media  selection  program.  This 
capability  would  be  similar  to  that 
used  in  the  Learning  Objectives 
Classification  (LOG)  Tool.  This  job 
aid  would:  (1)  standardize  the 
assignment  process  using  keywords,  (2) 
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provide  an  audit  trail  of  assignments, 
and  (3)  increase  the  accuracy  and 
speed  of  assignment.  Assignment  to 
both  categories  and  attributes  would 
be  provided  by  this  capability. 

The  second  assignment  approach  uses 
learning  phases  to  create  behaviorial 
statement  categories.  The  AFM  50-2 
(1979)  model  was  the  only  multiple 
form  model  that  uses  this  approach. 

No  automated  capability  is  deemed 
feasible  to  meet  the  requirements  of 
this  approach.  Accordingly,  a  manual 
assignment  method  is  recommended  for 
assigning  learning  phase  categories. 

The  preliminary  media  selection  model 
design  includes: 

(1)  a  decision  aid  that  will 
assist  the  user  in  selecting  the  most 
appropriate  media  selection  model; 

(2)  a  data  set  development 
capability  that  will  permit  the 
tailoring  and  management  of  the 
behavioral  statements  to  be  analyzed; 

(3)  a  media  selection  model 
development  function  that  will  provide 
for  the  entry  of  a  new  media  selection 
model  or  the  modification  of  a  model 
already  in  the  system;  and 

(4)  a  media  selection  model 
application  function  that  includes 
model  selection,  behavioral  statement 
assignment,  initial  media 
determination,  training  system 
alternatives  determination,  final 
media  selection,  media  results 
clustering,  and  course  media 
requirements  determination. 
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CD-ROM:  A  Delivery  Medium  for  CBT! 

Allen  L.  Luettgen,  F.  Kay  Houghton,  and  Andrew  E.  Andrews 

Computer-based  training  (CBT)  development  has  evolved  fromelectronic  page  turners  to 
a  sophisticated  instructional  environment.  The  use  of  numerous  large  image  files,  large 
digital  audio  files,  and  complex  computer- generated  graphics  files  places  great  demands 
on  a  system's  capacity  for  binary  storage  (disk  space).  One  solution  is  the  use  of  compact 
disc -read  only  memory  (CD-R(3M). 

The  explosive  growth  of  CD-ROM  players  in  the  marketplace  makes  CD-ROM  a  viable 
delivery  medium  for  CBT.  Recently,  a  CBT  package  for  radiation  protection  technicians 
(RPTs)  at  Los  Alamos  National  Laboratory  (LANL)  was  produced  on  CD-  ROM.  The 
course  is  delivered  on  a  multimedia  system  consisting  of  the  following;  MS  DOS-based 
computer,  CD-ROM  player,  high-resolution  video  graphics  array  (VGA)  monitor, 
scanned  color  images  with  graphic  overlays,  digital  audio,  and  mouse  interface. 

This  paper  will  allow  the  reader  to  assess  the  appropriateness  of  CD-ROM  for  a  specific 
project,  report  on  lessons  learned  from  the  RPT  project,  demonstrate  that  CD-  ROM  is 
within  reach  of  the  average  CBT  developer,  and  provide  guidelines  for  successfully 
developing  CD-ROM  based  CBT.  This  paper  can  serve  as  a  basic  primer  on  how  to 
adapt  CD-ROM  as  a  CBT  medium.  Some  of  the  technical  aspects  that  are  discussed  are 
listed  here:  CD-ROM  basics  for  CBT  development,  availability  of  low-cost  development 
environments,  file  organization,  optimization  of  CD-ROM  response  times,  effective  use 
of  digital  audio  and  still  frame  graphics,  and  animation  with  CD-ROM. 

CD-ROM  can  make  complex  CBT  possible  in  an  affordable  environment.  By  creatively 
incorporating  scanned  images,  laige  computer-generated  graphics,  and  digital  audio,  one 
can  avoid  the  expense  and  limitations  of  the  video  disc  technology. 
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CD-ROM:  A  DELIVERY  MEDIUM  FOR  CBT! 

Allen  L.  Luettgen,  Kay  Houghton,  and  Andrew  E.  Andrews 


Introduction  Computer-based  training 
(CBT)  development  has  evolved  from 
electronic  page  turners  to  a  sophisticated 
instructional  environment.  Today,  CBT 
packages  can  incorporate  scanned 
images,  large  computer-generated 
graphics,  and  dip^  audio.  The  use  of 
numerous  large  image  files,  large  digital 
audio  files,  and  complex  computer¬ 
generated  graphics  files  places  great 
demands  on  a  system's  capacity  for 
binary  storage  (disk  space).  When  using 
interactive  video  disc  is  prohibitive 
because  of  cost,  availability  of  delivery 
systems,  or  inadequate  distribution 
systems,  alternative  media  are  required. 
One  possibility  is  compact  disc-read 
only  memory  (CD-ROM). 

The  explosive  growth  of  CD-ROM 
players  in  the  marketplace  makes  CD- 
ROM  a  viable  delivery  medium  for 
CBT.  Recently,  a  CBT  package  for 
radiation  protection  technicians  (RPTs) 
at  Los  Alamos  National  Laboratory 
(LANL)  was  produced  on  CD-ROM, 
and  the  course  is  delivered  on  a 
multimedia  system. 

This  paper  will  allow  the  reader  to  assess 
the  appropriateness  of  CD-ROM 
for  a  specific  project,  report  on  lessons 
learned 

from  the  RPT  project,  demonstrate  that 
CD-ROM  is  within  reach  of  the  average 
CBT  developer,  and  provide  guidelines 
for  successfully  developing  CD-ROM- 
based  CBT. 

Why  CD-ROM  For  many  reasons,  CD- 
ROM  should  be  considered  for  the 
development  of  CBT.  CD-ROM  can 
make  complex  CBT  possible  in  an 
affordable  environment.  The  robustness 
of  CD-ROM  allows  a  person  to 
creatively  incorporate  scanned  images, 
large  computer-generated 
graphics,  and  digital  audio  into  training 
and  learning  applications.  Also,  the 


large  storage  capacity  of  the  CD-ROM 
will  allow  it  to  play  a  vital  role  in  future 
CBT  and  learning  applications. 

Cost  In  the  last  five  years,  the  cost  of 
CD-ROM  production  and  development 
has  decreased  so  that  is  within  the  reach 
of  many  training  organizations.  CD- 
ROM  is  now  a  cost-effective  delivery 
vehicle  for  (DT.  When  considering  the 
use  of  CD-ROM  for  CBT  delivery,  one 
should  consider  the  costs  of 
development  and  production. 

Production  Costs  The  cost  of  CD- 
ROM  production  includes  the  following 
items:  development  system  equipment, 
deliveiy  system  equipment,  and 
mastering  or  production  of  the  CD- 
ROM.  The  equipment  requirements  for 
the  delivery  system  and  the  development 
system  are  different.  Because  the 
development  system  requirements  are 
dependent  on  ^e  delivery  system 
requirements,  we  will  discuss  the 
delivery  system  requirements  first. 

Several  technology  options  must  dc 
considered  when  deriving  the 
specifications  of  the  delivery  system. 
The  first  technology  consideration  is  the 
selection  of  the  delivery  platform,  which 
is  based  on  cost,  availability,  and 
performance  considerations.  A  bare- 
bones  80386  PC  system,  which  costs 
less  than  $1,200,  was  determined  to  be 
the  tnost  appropriate  platform  for  the 
RPT  Trainer.  A  less  capable  machine  is 
not  recommended  because  of  the 
performance  limitations  of  the  CD-ROM 
unit.  The  faster  data  transfer  rates  of  the 
80386  reduces  the  amount  of  time  it 
takes  to  move  the  data  from  the  MS- 
DOS(TM)  buffers  to  program  memory, 
thus  blowing  the  CD-ROM  to  transfer 
data  at  its  maximum  capacity. 

Another  consideration  is  the  use  of  color 
displays.  Because  the  cost  of  color  and 
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monochrome  presentations  is  virtually 
the  same,  the  decision  to  use  color  or 
monochrome  should  be  based  on  CBT 
design  considerations.  Because  there  is 
little  difference  in  cost,  we  recommend 
using  color  displays  that  allow  both 
color  graphics  and  scanned  images. 

Most  CBT  projects  include  the  use  of 
graphics,  and  the  cost  of  adding  graphics 
to  the  delivery  system  is  negligible. 
Virtually  all  display  boards  have 
graphics  capabilities.  Furthermore, 
improvements  in  technology  have  made 
the  cost/performance  relationship 
between  Video  Graphics  Array  (VGA) 
and  older  standards  comparable.  With 
comparable  cost/performance  ratios, 

VGA  is  strongly  recommended. 

One  of  the  most  important  features  of  a 
CBT  project  is  the  user  interface. 

Sevet^  options  are  available  for  an  input 
pointing  device.  The  mouse  is  quickly 
becoming  the  preferred  pointing  device. 
Nevertheless,  light  pens  and  touch 
screens  are  available  options  but  are  not 
directly  supported  by  VGA  and  require 
additional  equipment,  thus  increasing  the 
cost  of  the  delivery  system. 

During  the  design  of  the  CBT,  a  decision 
must  be  made  to  use  audio  or  not.  The 
cost  of  adding  audio  to  a  CBT  depends 
c  n  whether  analog  or  digital  audio  is 
u:ed.  An  analog  system  includes  the 
tape  player  and  the  speakers;  the  price  of 
an  analog  audio  system  starts  at  $50  and 
depends  on  the  required  fidelity.  A 
digital  audio  system  includes  a  digital 
audio  board  and  speakers.  The  board  we 
used,  the  DSA-320  audio  board  from 
OnLine,  Inc.(TM),  is  about  $400.  Other 
boards  and  playback  systems  are 
advertised  for  less  money,  which 
indicates  that  the  cost  of  digital  audio 
boards  is  decreasing. 

The  CD-ROM  player  ($400  to  $800)  is 
required  for  a  CD-ROM  CBT.  The 
International  Standards  Organization 
(ISO)  adapted  the  ISO  99^  as  the 
standard  that  defines  a  CD-ROM 
directory  structure.  Therefore,  to  ensure 
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that  the  CD-ROM  unit  can  read  any  CD- 
ROM  disc,  the  unit  should  be  able  to 
read  the  ISO  9960  CD-ROM  disc 
format 

Once  the  delivery  system  has  been 
defined,  the  development  system 
requirements  can  be  derived  from  the 
delivery  system.  The  development 
system  specifications  are  those  of  the 
^livery  system  plus  a  read/write 
medium.  Because  a  training  project  is 
revised  numerous  times  before  it  is 
mastered,  the  development  system  needs 
to  easily  accommodate  changes  and  yet 
simulate  the  CD-ROM,  which  requires  a 
large  read/write  disc  storage. 

The  expense  of  a  read/write  medium 
varies.  Listed  below  are  costs  per 
megabyte  of  some  selected  media 
(Dataware,  Inc.,  1989): 

removable  hard  $15 

disk 

fixed  hard  disk  $10 

magneto-optical  $10 

removable  disk 

Of  the  three  media  listed,  we 
recommend  the  magneto-optical 
removable  disc  because  it  has 
performance  characteristics  most  similar 
to  those  of  the  CD-ROM. 

The  mastering  costs  of  a  CD-ROM 
include  formatting  the  data  to  be 
mastered  and  the  mastering  expense. 
Formatted  media,  magneto-optical 
cartridge  or  9-track  tape  are  sent  to  the 
mastering  company  (3M).  Using  a 
magneto-opticd  disc  is  recommended 
because  the  production  of  a  9-track  tape 
requires  additional  equipment. 

Dataware,  Inc.  (1989)  estimates  the  cost 
of  mastering  a  disc  to  be  $1,500  and  the 
cost  of  each  disc  to  be  about  $2. 

Development  Costs  As  in  all  CBT 
development,  the  costs  are  directly 
proportional  to  the  complexity  and  size 
of  the  training  system  (Gery,  1989).  The 


complexity  includes  the  quality  and 
quantity  of  graphics,  text,  and  audio 
used,  as  well  as  the  intricacy  of  the 
feedback  branching  structure. 
Fortunately,  the  CD-ROM  can  help 
reduce  the  costs  of  program  management 
and  maintenance. 

The  program  management  costs  are 
reduced  by  the  use  of  MS-DOS  CD- 
ROM  Extensions(TM)  file  structure.  On 
the  PC,  the  developer  perceives  this  file 
structure  to  be  a  MS-DOS  (TM)  file 
system,  which  allows  the  use  of 
(hrectories  and  subdirectories.  Using 
directories  and  subdirectories  permits 
convenient  program  modularization  and 
integration  of  the  modules. 

One  of  the  main  concerns  of  CBT 
development  is  the  updating  of  the 
training  material.  CD-ROM  file 
structure  allows  easy  updating  of 
individual  files.  Because  CD-ROM  is 
read-only,  the  maintenance  of  the 
training  program  is  limited  to  the  costs 
associate  with  mastering  of  a  new  CD- 
ROM  (~$ 1,500)  plus  m^flcation  of 
the  program. 

Media  Robustness.  A  CD-ROM  is 
difficult  to  copy,  write  protected,  crash- 
resistant,  large  in  capacity,  an  easy 
medium  for  delivery  of  CBT,  and 
physically  durable,  making  it  a  robust 
medium.  Because  of  the  volume  of  CD- 
ROM  data,  the  CD-ROM  is  difficult  to 
copy. 

Write  Protection  Because  the  CD- 
ROM  is  a  read-only  device,  it  is 
impossible  for  anyone  to  change, 
intentionally  or  unintentionally,  any  data 
on  the  medium.  The  read-only 
characteristics  of  the  CD-ROM  have 
many  benefits.  For  example,  we 
delivered  a  CBT  course  on  a  traditional, 
magnetic  hard  disc  to  the  U.  S.  Army . 
The  students  began  adding  unauthorized 
enhancements  to  the  courseware. 

Several  times  the  enhancements  forced 
us  to  reinstall  the  entire  courseware. 
Preventing  the  possibility  of  the  students' 


modifying  the  courseware  in  any  form 
helps  ensure  some  training  consistency. 

Crash  Resistant  The  medium  is  crash 
resistant  because  a  mechanical  head  does 
not  pass  over  media.  The  information  is 
read  when  a  movable  miiror  reflects  a 
low-power  laser  onto  the  media, 
eliminating  the  possibility  of  a  head 
crash  on  the  media. 

Large  Capacity  The  large  capacity, 

640  Mbytes,  of  the  CD-ROM  allows  an 
entire  training  package,  or  possibly 
several  training  pacl^ges,  to  be  resident 
on  a  single  optical  disc  instead  of  dozens 
or  hundi^  of  floppy  discs.  Therefore, 
the  requirement  of  transferring  sections 
of  the  program  to  and  from  the  hard  disc 
is  eliminated.  Also,  the  large  capacity 
reduces  the  amount  of  actu^  disc 
swapping  required,  thus  reducing  the 
chances  of  d^aging  the  media. 

Easy  Delivery  The  CD-ROM  disc  is 
similar  to  a  floppy  disc  because  it  is 
removable.  The  delivery  is  simpliHed 
because  the  CD-ROM  requires  only  the 
insertion  of  a  single  disc  instead  of 
copying  many  floppy  discs.  In  addition, 
the  CD-ROM  has  a  large  data  capacity; 
therefore,  it  is  possible  to  include 
original  source  code  and  supplemental 
software  when  the  system  is  distributed. 
This  medium  also  eliminates  the  need 
for  back-up  copies  because  the  original 
data  cannot  be  corrupted  by 
unintentionally  overwriting  the  update 
disc. 

Physically  Durable  The  CD-ROM  is 
physically  a  rather  tough  medium.  It  is 
composed  of  a  vinyl-type  material, 
which  should  not  touched  by  the  user. 
If  the  user  inadvertently  touches  the 
medium,  it  will  not  necessarily  be 
destroyed.  Even  though  it  should  be 
kept  in  an  environment  that  is  as  dust 
fhte  as  possible,  it  can  be  operated  in  an 
office  without  any  special  precautions. 

Future  Applications  The  opportunities 
for  future  use  of  CD-ROM  technology 
are  very  exciting.  The  potential  for 
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storing  information  is  almost  boundless. 
With  one  CD-ROM  disc  containing 
approximately  the  same  amount  of  data 
as  150,000  to  200,000  pages  of  text  or 
1,200  to  1,500  floppy  discs  (Bitter, 
1988),  the  ability  to  have  large 
instructional  structures  is  possible.  A 
few  of  the  applications  that  can  benefit 
from  the  use  of  CD-ROM  are  described 
below. 

Hypertext  The  extensive  use  of 
hypertext  structures  is  feasible  due  to  the 
large  capacity  of  the  CD-ROM. 
Hypertext  stmctures  are  based  on 
nonlinear  access  of  information.  Instead 
of  progressing  through  data  in  a  step-by- 
step  fashion,  hypertext  allows  the  user  to 
jump  from  one  segment  to  another. 
Hypertext  can  be  roughly  compared  with 
reading  a  book  and  then  using  the  index 
to  locate  more  information.  Hypertext 
allows  the  netwoiicing  of  many  large 
structures. 

Simulation-based  Training  Limited 
space  has  made  the  use  of  computer- 
based  simulations  in  training  packages 
difficult  to  write.  To  be  realistic, 
simulations  need  to  be  dynamic,  that  is, 
respond  to  learner  input  in  a 
sophisticated  enough  manner  to  consider 
the  impact  of  an  action  on  the  total 
simulation  system.  To  be  effective,  the 
simulation  should  contain  as  many 
variables  as  possible.  As  the  simulated 
system  grows  and  becomes  more 
complex,  the  number  of  possible  paths 
and  outcomes  increases  dramaticdly 
(often  factorially),  and  the  large  storage 
capacity  of  the  CD-ROM  allows  the 
storage  of  these  complex  systems. 
Finally,  incorporating  simulation  into 
CBT  in  a  manner  that  fully  leverages  the 
benefits  of  the  simulation  requires  a 
different  CBT  paradigm  that  again 
increases  data  storage  demands 
(Spangenberg,  1989). 

Available  References  The  CD-ROM 
can  easily  incorporate  manuals  and  other 
references  that  are  available  to  the 
learner,  allowing  greater  learner  control 
with  very  little  programming  overhead. 


Often  while  performing  a  job,  the 
employee  uses  manuals  and  references. 
The  ability  to  access  these  manuals 
during  training  makes  the  training  more 
realistic,  which  is  especially  important 
during  performance-based  training. 

Intelligent  Tutor  The  large  capacity 
allows  the  program  to  contain  a  greater 
number  of  possible  interactions.  This 
capacity  can  be  used  in  more 
sophisticated  intelligent  tutoring 
systems.  A  tutoring  system  based  on 
artificial  intelligence  (AI)  will  be  able  to 
access  a  large  data  base  of  stored 
knowledge  and  make  decisions  based  on 
that  knowledge.  Gary  Bitter  (1988) 
contends  that  in  our  lifetime  intelligent 
tutoring  systems  will  be  able  to  identify 
students'  weaknesses,  evaluate  their 
needs  and  then  tutor  the  students. 
Therefore,  the  intelligent  tutor  can  do 
even  more,  can  evaluate  the  individual's 
learning  style,  and  can  tailor  the  lesson 
accordingly.  However,  as  the  capacity 
of  the  program  grows,  so  does  its  size. 

A  Place  in  DVI  World  The  CD-ROM 
has  a  place  in  the  Digital  Video 
Interactive  (DVI)  world.  DVI  converts 
analog  audio  and  full  motion  video  data 
into  a  digital  form  at  a  rate  of  30  frames 
per  second.  A  single  frame  is 
represented  digitally  by  appropriately 
750  kbytes  of  data.  Compression 
techniques  help  reduce  the  amount  of 
storage  needed  to  generate  a  full  motion 
sequence  on  a  computer.  Even  with 
compression  rates  of  50  to  1,  one  second 
of  video  requires  450  kbytes  of  storage. 
Updates  to  a  DVI  database  can  entail 
tens  to  hundreds  megabytes  of  data.  The 
large  data  capaci^  of  the  CD-ROM 
ma^es  it  the  medium  of  choice  for  DVI 
updates. 

This  paper  does  not  advocate  the 
adoption  of  DVI  over  interactive  video 
disc  (rVD).  The  issue  of  the  need  for 
full  motion  video  in  CBT  is  linked  to 
instructional  design  and  cost-benefit 
considerations.  While  the  use  of  still 
frame  graphics  and  digital  audio 
(possible  with  CD-ROM)  can  be  argued 
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as  a  cost-attractive  alternative  to  IVD, 
comparing  FVD  with  DVI  is  beyond  the 
scope  of  Ais  paper.  It  should  be  noted, 
however,  that  adoption  of  CD-ROM  is 
an  investment  compatible  with  the  DVI 
technology. 

Training  and  learning  applications  are 
limited  only  by  one's  imagination,  not  by 
the  storage  capabilities  of  the  system. 
Thus,  CD-ROM  is  playing  a  significant 
role  in  charting  the  future  of  computer- 
based  training. 

Developing  CBT  for  a  CD-ROM  The 
developmental  process  of  a  CBT  for 
CD-ROM  delivery  is  similar  to 
developing  CBT  for  hard  disc  delivery. 
However,  during  development,  a  person 
must  consider  the  unique  characteristics 
of  the  CD-ROM;  that  is,  it  is  a  read-only 
device  with  slower  access  time  and  has  a 
larger  capacity  than  that  of  a  hard  disc. 

Authoring  Languages  During  the 
development  of  the  RPT  Trainer,  we 
used  two  different  software 
methodologies  for  developing  lessons. 
The  first  module  was  developed  using 
the  OASYS(TM)  authoring  system,  and 
the  second  was  developed  using  the 
high-level  programming  language 
Modula-2.  Both  modules  were 
developed  for  CD-ROM  production.  As 
far  as  the  CD-ROM  was  concerned, 
there  was  no  advantage  to  either  system. 
However,  because  a  programming 
language  allows  more  complex  data 
structures  and  branching  structures  than 
an  authoring  tool,  it  is  possible  to  exploit 
the  large  capacity  of  the  CD-ROM  more 
completely  with  the  programming 
language. 

Implications  of  a  Read-Only  Device 
Because  a  CD-ROM  is  a  read-only 
device,  files  cannot  be  written  to  it.  For 
example,  student  records  must  be  kept 
on  either  the  resident  hard  disc,  a  floppy 
disc,  or  a  network  file  server,  which  can 
cause  some  problems  for  an  authoring 
system. 


Because  a  CD-ROM  is  a  read-only 
device,  multiple  mastering  during  CBT 
development  would  be  prohibitively 
expensive.  A  cost-effective  alternative 
to  multiple  mastering  is  the  simulation 
of  a  CD-ROM  on  a  h^ard  disc  or  a 
magneto-optical  drive. 

Hard  disc  simulation  of  the  CD-ROM  is 
not  recommended  for  two  reasons.  First, 
if  a  hard  disc  is  used,  the  data  would 
have  to  be  transferred  to  magnetic  tape 
before  mastering  a  CD-ROM.  This  step 
incurs  an  added  expense  for  the 
magnetic  tape  equipment.  Second,  the 
peiformance  characteristics  of  the  hard 
disc  and  the  CD-ROM  are  substantially 
different.  The  CD-ROM  performance  is 
significantly  slow  when  compared  with 
that  of  the  hard  disc. 

A  magneto-optical  disc  is  a  better 
simulator  of  a  CD-ROM  than  a  hard 
disc,  and  the  capacity  and  performance 
of  the  magneto-optical  disc  is 
comparable  to  that  of  the  CD-RCM. 

The  magneto-optical  cartridge  has  two 
sides;  each  side  has  half  the  capacity  of 
that  of  a  CD-ROM.  If  a  removable 
magneto-optical  disc  is  used,  the 
cartridge  can  be  sent  to  the  mastering 
company,  eliminating  transferring  the 
system  to  magnetic  tape.  As  a  point  of 
comparison,  when  we  compared  the  cost 
of  magneto-optical  disc  with  the  cost  of 
magnetic  tape  equipment,  the  cost  was 
$5,000  versus  $30,000.  At  $5,000  for  a 
magneto-optical  disc  system,  CD-ROM 
premastering  is  affordable. 

Performance  considerations  When 
developing  CBT  for  CD-ROM  delivery, 
the  peiformance  of  the  CD-ROM  needs 
to  be  taken  into  account.  The  slower 
access  time  and  larger  capacity  are  the 
two  conditions  that  are  unique  to  CD- 
ROM. 

Slow  Access  Time  The  performance 
characteristics  are  defined  in  terms  of 
the  CD-ROM  drive  performance  and  the 
operational  characteristics  of  the 
software  driver  interfacing  to  the  CD- 
ROM  drive. 
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The  CD-ROM  drive  performance 
characteristics  include  the  data  transfer 
rates  and  access  time.  The  data  transfer 
rates  are  ISO  kbytes/s  sustained  and  600 
kbytes/s  burst.  Access  time  is  0.8 
seconds  for  a  full  stroke  seek.  Seek  time 
is  the  amount  of  time  necessary  to  move 
the  mirror  of  the  laser  beam  to  the 
correct  position  on  the  CD-ROM.  An 
average  stroke  is  one-third  of  a  full 
stroke,  thus  requiring  0.5  seconds.  The 
typical  rotational  speeds  of  a  CD-ROM 
are  530  rpm  at  the  innermost  track  with 
a  constant  linear  velocity  of  1.4  meters/s 
and  200  rpm  at  the  outmost  track  with  a 
constant  linear  velocity  of  1.2  meters/s 
(Sony  Corporation  1988). 

The  operational  characteristics  of  the 
software  driver  interfacing  to  the  CD- 
ROM  drive  can  have  a  large  impact  on 
the  drive  performance.  The  software 
driver  commands  the  mirror  to  move  and 
read  data  from  the  disc.  The  position  of 
the  mirror  after  the  read  operation  can 
greatly  affect  the  overall  performance  of 
die  next  read.  The  MS-E)OS  CD-ROM 
Extensions(TM)  leaves  the  mirror  at  the 
last  seek/read  position.  The  next 
seek/read  will  require  only  minimal 
mirror  movement  to  read  adjacent  data. 

Slower  performance  time  of  the  CD- 
ROM  could  have  a  dramatic  impact  on 
disc-intensive  simulations.  To  help 
alleviate  most  of  this  problem,  two 
approaches  are  recommended.  The  fu-st 
involves  disc  caching  of  the  data,  and 
the  second  is  intelligent  file  organization 
on  the  disc. 

Disc  Caching  of  Data  Disc  caching  of 
the  data  on  a  CD-ROM  can  be 
accomplished  by  two  methods.  The  first 
method  is  quite  simple.  During 
installation  of  the  software  driver  for  the 
CD-ROM,  one  selects  the  largest 
possible  buffer  size  allowed.  The 
maximum  size  of  the  buffer  is  driver 
dependent,  but  generally  it  is  64  kbytes. 
This  buffer  is  used  as  temporary  storage 
of  data  from  the  CD-ROM.  Sectors 
from  the  CD-ROM  are  read  until  the 


buffer  is  full,  thus  taking  advantage  of 
the  maximum  transfer  rate.  Only  the 
data  requested  are  transferred  fi'om  the 
buffer  to  the  program. 

The  second  method  requires  the 
developer  to  buffer  files  from  the  CD- 
ROM  into  DOS  memory  before  they  are 
used.  The  developer  must  track  the  files 
buffered.  This  method  requires  a 
significant  amount  of  progranuning 
overhead  and  inconvenience  on  the  part 
of  the  developer.  Therefore,  this  method 
should  not  be  used  unless  absolutely 
necessary.  However,  this  method  may 
be  required  for  disc-intensive 
simulations.  Because  this  method  is  too 
difficult  for  the  novice  user,  it  should  be 
implemented  only  by  the  experienced 
programmer. 

File  Organization  The  intelligent  file 
organization  is  related  to  the  operational 
characteristics  of  the  software  driver.  If 
the  driver  is  the  MS  DOS  CD-ROM 
Extension(TM),  then  placing  files  into 
subdirectories  and  locating  related 
subdirectories  together  on  CD-ROM  is 
recommended.  Organizing  files  into 
subdirectories  allows  the  mastering 
company  to  place  all  files  of  the 
subdirectory  sequentially  on  the  disc. 
Thus,  minimum  mirror  movement 
(access  time)  is  required  to  read 
subsequent  data  in  this  subdirectory. 
Specifying  the  order  of  files  within  a 
subdirectory  increases  performance 
minutely;  therefore,  the  subdirectory 
specification  is  the  lowest  data 
placement  specification  required  for  the 
CD-ROM.  Because  CD-ROM  has 
nearly  constant  linear  velocity  anywhere 
on  the  disc,  placement  of  the 
subdirectories  relative  to  the  inner  or 
outer  tracks  does  not  matter. 


Lessons  Learned  Several  lesson  were 
learned  during  development  of  the  RPT 
Trainer.  This  section  deals  with  specific 
lessons  learned  with  our  hardware  and 
software  suites.  These  lessons  included 
dealing  with  scanned  graphics,  teaching 
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techniques,  working  with  audio, 
optimizing  seek  times  of  files,  and 
working  with  a  magneto-optical  disc. 

Digitized  Images  A  scanned  graphic  is 
a  picture  that  has  been  digitized  into  a 
computer  representation;  all  graphics 
were  bit  map  graphics.  Digitized  images 
consist  of  any  type  of  picture  not  created 
with  a  paint  package.  Furthermore, 
these  images  could  not  be  direcdy 
displayed  by  the  VGA  subsystem. 

Digital  Image  Production  The  process 
of  producing  a  digital  image  begins  with 
capturing  a  picture  from  a  video  tape,  a 
still  photograph,  or  a  live  camera  and 
then  converting  it  to  a  displayable 
format.  This  process  is  described  below. 

Still  Pnotograph  Image  Capture  Still 
photographs  are  converted  into  a 
computer  representation  by  a  scanner. 
We  used  a  color,  300-dot-per-in. 
Howtech(TM)  sc^mner.  The  scanner 
places  the  digital  representation  into  a 
file.  The  resolution  of  the  captured 
picture  is  512  x  512.  The  file  format  of 
the  scanned  picture  is  compatible  with 
the  Truevision  Targa-16(TM)  image 
capture  and  display  system.  Because  the 
output  file  format  used  the  Truevision 
'rarga-16(TM)  file  format,  the  digital 
representation  contained  a  maximum  of 
32,768  colors.  We  experienced  no 
problems  with  the  scanner  during  the 
courseware  development. 

Video  Tape  or  Live  Camera  Image 
Capture  A  single  video  tape  frame  or  a 
snapshot  from  a  live  camera  is  convened 
into  a  computer  representation  of  the 
image  by  a  capture  and  display  system. 
We  used  the  Truevision  Targa-16(TM) 
system,  which  places  the  digital 
representation  of  the  image  into  a  file. 
The  resolution  of  the  captured  picture  is 
512  X  j  12,  and  the  maximum  number  of 
colors  is  32,768. 

Because  our  VCR  did  not  have  still 
frame  capability,  the  capturing  of 
pictures  from  video  tape  was  an  arduous 
task.  The  technician  had  to  use  a  trial 


and  error  method  to  capture  the  correct 
picture  frame.  We  smongly  recommend 
the  use  of  a  VCR  with  still  frame 
capability. 

Picture  Conversion  After  the  picture  is 
captured  from  a  still  photograph,  video 
tape,  or  live  camera,  it  has  to  be 
convened  into  a  format  displayable  by 
the  VGA  system,  which  means  that  the 
number  of  colors  of  the  picture  must  be 
reduced.  The  VGA  system  is  capable  of 
displaying  16  colors  in  the  640  x  480 
resolution  mode.  The  instructirnal 
strategies  used  in  the  RPT  Tr  ler 
further  limited  the  number  of  available 
colors  to  12.  Thus,  the  conversion 
process  entails  converting  the  digital 
image  with  32,768  colors  into  a  picture 
containing  only  12  colors. 

Limitation  of  the  VGA  Standard  A 
brief  discussion  of  how  pictures  are 
drawn  on  the  screen  is  required  to 
understand  the  source  of  limitations 
number  of  colors  displayabL  un  the 
screen.  We  will  try  to  make  this 
discussion  as  painless  as  possible. 

Every  picture  uses  the  colors  from  a 
color  map  to  draw  itself  on  the  screen 
(this  is  a  requirement  of  VGA  sLindard). 
A  color  map  is  a  palette  of  colors,  and 
the  picture  selects  the  most  appropriate 
colors  by  defining  the  set  of  colors  in  the 
color  map. 

The  VGA  standard  liriirs  the  number  of 
colors  allowed  in  the  color  palette  and 
screen  resolutions.  (A  screen  resolution 
is  the  maximum  number  of  dots  in  the 
hv'rizontal  and  vertical  direction.  These 
dots  are  called  pixels.)  Our  courseware 
used  two  different  screen  resolutions, 
640  X  480  and  320  x  200  (height  and 
width)  pixels.  Both  screen  '.esolutions 
fill  the  s'lme  area  on  die  screen  but  the 
lower  n.sol  Jtion  (320  x  200)  uses  a  pixel 
that  is  fc’ir  times  larger  than  the  higher 
resoluti'  -n  pmei  size. 

The  VGA  standard  states  that  the  320  x 
200  resolution  can  use  256  colors  and 
the  640  X  480  resolution  ean  use  only  16 


colors.  This  standard  is  based  on  the 
capabilities  and  amount  of  RAM 
available  on  the  VGA  card.  The 
baseline  VGA  card  has  256  kbytes  of 
RAM.  In  the  640  x  480  resolution 
mode,  4  bits  (1/2  byte)  are  needed  to 
define  16  colors  (16  =  2  raised  to  the  4th 
power).  Because  each  pixel  needs  only 
one-half  byte,  two  pixels  can  be  stored 
in  a  single  byte.  The  amount  of  RAM 
needed  to  define  a  640  x  480  picture 
with  1 6  colors,  using  2  pixels  per  byte,  is 
153,600  bytes  or  153.6  kbytes. 


640  X  480  =  307,200  pixels  per  screen. 

1  bytes  =  2  pixels. 

307,200  pixels  x  (1  byte/2  pixels)  = 
153,600  bytes  (or  153.6  kbytes). 


If  256  colors  are  used  in  a  640  x  480 
resolution  mode,  8  bits  (1  byte)  are 
needed  to  define  256  colors  (256  =  2 
raised  to  the  8th  power).  The  amount  of 
RAM  needed  to  define  a  640  x  480 
picture  with  256  colors,  using  1  pixel  per 
byte,  is  307,200  bytes  or  307.2  kbytes. 


640  X  480  =  307,200  pixels  per  screen. 

1  bytes  =  1  pixels. 

307,200  pixels  x  (1  byte/1  pixel)  = 

307,200  bytes  (or  307.2  kbytes). 


Hence,  with  only  256  kbytes  of  memory 
available  on  the  standard  VGA  card, 
only  pictures  with  640  x  480  resolution 
with  16  colors  are  displayable.  The 
fewer  total  number  of  pixels  used 
(number  of  pixels  wide  times  number  of 
pixels  high)  in  a  picture,  the  grainier  the 
picture  appears.  The  use  of  more  colors 
helps  to  visually  compensate  for 
increased  graininess.  As  the  detail  of  the 
picture  being  displayed  increases,  a 
higher  screen  resolution  is  required. 
Trade-offs  have  to  be  made  among 
displayed  detail,  the  color  quality  of  the 


picture  and  using  a  VGA  card  with  more 
memory.  We  generally  used  the  higher 
resolution  to  display  the  picture  because 
the  detail  of  the  picture  was  more 
important  than  color  quality.  Also, 
changing  between  screen  resolutions 
causes  a  disconcerting  flash  on  the 
monitor.  The  alternative  of  using  a 
VGA  card  with  256  kbytes  of  RAM 
memory  was  not  available  to  us. 

Limitation  Imposed  by  Learning 
Strategies  Learning  strategy  limitations 
begin  to  tread  on  the  choice  of  teaching 
techniques  or  styles.  The  most 
appropriate  learning  strategy  will  not  be 
addressed  in  this  paper,  only  the 
limitations  imposed  on  the  scanned 
graphics  by  our  selected  learning 
strategies  technique  will  be  described. 
We  used  a  learning  strategy  based  on 
learner  control,  which  allows  the  learner 
to  move  freely  from  one  part  of  the 
program  to  another.  To  accomplish  this 
mobility,  a  command  bar  appeared  on 
every  screen.  The  command  bar 
consisted  of  user  options,  such  as  repeat 
audio,  help,  exit,  continue,  and  so  on. 

For  consistency,  the  command  bar  was 
displayed  using  the  same  three  colors. 
Furthermore,  each  screen  used  the  same 
background  color,  which  imposed  a 
limitation  that  every  screen  and  every 
color  map  would  contain  the  same  four 
colors  in  the  same  positions  of  the  color 
map.  The  limitation  of  fixing  four  of  the 
available  16  colors  significantly 
curtailed  the  flexibility.  If  256  colors 
were  used,  the  penalty  imposed  by  the 
four  fixed  colors  would  have  been 
insignificant.  A  640  x  480  screen  with 
256  colors  requires  an  image  file  of 
307.2  kbytes,  which  would  consume  the 
capacity  of  a  hard  disc  quickly,  but 
almost  2,000  such  images  can  be  stored 
on  a  single  CD-ROM. 

Conversion  Process  The  captured 
image  contains  32,768  colors,  far  more 
then  the  VGA  standard  allows.  The 
conversion  process,  which  reduces  the 
number  of  coIots  in  the  digital  image 
representation  to  16  or  256  colors, 
requires  two  steps.  The  first  step  uses 


the  commercial  image  conversion 
package  TEGA(TM)  from  Videotex 
Systems  to  convert  the  32,768  colors 
into  16  or  256  colors.  After  the  picture 
is  in  a  format  compatible  with  the  VGA 
standard,  the  four  fixed  colors  are 
changed  by  a  manual  process. 

During  the  manual  process,  the  picture 
was  displayed  in  a  paint  package,  and 
the  four  least  obtrusive  colors  were 
changed  to  the  RPT  Trainer  "standard 
colors."  Changing  four  colors  out  of  256 
colors  was  an  easy  process,  but  changing 
four  colors  out  of  16  colors  required 
considerably  more  effort  and  a  good  eye. 
The  digital  image  was  then  ready  to  be 
displayed  as  part  of  the  RPT  Trainer  on 
the  VGA  screen. 

Digital  Image  Enhancement  The 
digital  images  used  in  the  CBT  greatly 
enhanced  the  training  by  allowing  the 
student  to  easily  visualize  the  equipment 
and  locations.  The  VGA  standard  and 
the  learning  strategies  imposed  rather 
severe  constraints  on  the  digital  images, 
and  the  conversion  process  involved  far 
too  much  manual  effort.  To  reduce  the 
amount  of  effort  required  to  conven 
digital  images  to  a  displayable  format, 
some  analysis  of  the  conversion  process 
was  performed. 

Analysis  of  Digital  Image  Conversion 
The  majority  of  the  effort  expended  and 
the  loss  of  visual  clarity  occurred  during 
the  color  reduction  process  The 
conversion  ot  die  digital  image  from 
32,678  colors  to  16  by  the  commercial 
software  introduced  the  largest  visual 
impairment  of  the  picture.  After  the 
software  conversion,  an  inordinate 
amount  of  time  and  skill  was  required  to 
change  the  four  additional  colors.  The 
visual  characteristics  of  the  image  were 
remarkably  better  when  the  number  of 
colors  was  increased  from  16  to  256,  and 
the  color  reduction  required  for  the 
command  bar  was  easily  accomplished. 
Some  exploration  of  improved  color 
content  at  higher  resolutions  began  near 
the  end  of  the  project,  but  too  late  to 
impact  the  developmental  process. 


Enhanced  Color  Resolution  Systems 
We  investigated  the  use  of  the  updated 
VGA  standi  (Super  VGA)  of  640  x 
480  resolution  with  256  colors  because  it 
is  supported  by  all  monitor  vendors.  A 
higher  resolution  was  not  explored 
because  it  would  necessitate  upgrading 
both  the  monitor  and  the  graphics  board. 
Because  VGA  monitors  are  analog 
monitors,  the  number  of  colors  displayed 
is  a  function  of  only  the  graphics  boani 

Choosing  the  Super  VGA  with  640  x 
480  and  256  colors  would  impose 
constraints  only  on  the  graphics  board, 
which  is  the  least  expensive  item  in  the 
video  subsystem.  Many  commercially 
available  graphics  boards  supporting  this 
Super  VGA  resolution  are  available. 

One  must  be  careful  when  specifying  a 
graphics  board  because  not  all  graphics 
boards  can  support  this  Super  VGA 
resolution  using  a  standard  VGA 
monitor.  We  recommend  that  one 
selects  a  graphics  board  that  supports 
this  Super  VGA  resolution  and  requires 
the  use  of  a  standard  VGA  monitor.  The 
VGA  monitor  standard  horizontal  sync 
rate  is  3 1 .64  kHz  and  the  VGA  standard 
vertical  sync  rate  is  70.08  Hz. 

Because  this  Super  VGA  resolution 
extension  to  the  VGA  standard  is  not  a 
standard,  the  implementation  from  one 
graphics  board  to  another  differs.  We 
found  some  similarities  among  these 
boards  and  developed  a  prototype 
graphics  driver  to  demonstrate  ±e 
interoperability  among  multiple  graphics 
boards.  Since  this  development  work 
was  completed,  commercial  drivers 
supporting  multiple  graphics  boards 
have  become  available  (such  as  PCX 
Programmer  Toolkit|TM]  from  Genus 
Microprogramming  [Genus 
Microprogramming,  1989]).  The 
increased  visual  clarity  and  the  ease  of 
the  developmental  process  are  strong 
motivating  factors  in  recommending  the 
higher  color  resolution  for  all  future 
projects.  Almost  all  graphics  boards  are 
supporting  the  640  x  480  x  256 
resolution.  Generally,  if  an  upgrade  is 


176 


required,  the  upgrade  entails  inserting 
video  RAM  chips  onto  the  graphics 
board.  This  process  requires  low  to 
modest  technical  abilities.  Furthermore, 
the  commercial  conversion  package 
TEGA(TM)  and  most  paint  packages 
support  tliis  Super  VGA  resolution. 

Inability  to  Animate  Bit  Map 
Graphics  One  disadvantage  of  using 
bit-mapped  pictures  was  the  inability  to 
animate  large  pictures.  Large 
animations  were  impossible  because  of 
the  time  required  to  seek  and  read  each 
segment  from  the  CD-ROM.  Also,  the 
large  animations  were  very  slow  and 
unrealistic.  Hence,  we  could  animate 
only  small  areas  of  the  screen,  such  as 
the  needle  on  an  instrument.  It  should 
be  noted  that  Intel  Corps'  DVI 
technology  essentially  is  a  full-screen 
animation  process,  but  at  greatly 
increased  cost. 

Audio  The  pairing  of  audio  with  text 
can  greatly  enhance  student  learning  and 
retention  (Buckner  and  McGrath,  1961). 
The  audio  can  be  a  paraphrasing  or 
additional  information;  simply  reading 
the  written  text  is  not  veiy  interesting  to 
the  student  or  to  the  developer.  Audible 
cues  can  be  given  to  the  student  without 
forcing  him  to  read  the  screen,  which 
can  reduce  eyestrain.  Furthermore,  the 
use  of  audio  can  lower  the  reading 
comprehension  level  required  of  the 
training  package.  Digit^  audio  has  been 
used  in  several  training  programs.  For 
example,  American  Express  is  using 
digital  audio  on  CD-ROM  to  train 
telephone  operators  (Videodisc  Monitor, 
1990),  and  igital  audio  is  used  with  the 
RPT  Trainer  at  Los  Alamos  National 
Laboratory. 

It  is  difficult  for  the  average  listener  to 
differentiate  between  the  two  basic  types 
of  audio:  analog  and  digital.  Analog 
audio  contains  a  continuous  range  of 
frequencies  (tones)  and  amplitudes 
(volume).  Examples  of  an^og  audio  are 
records  and  traditional  casette  tapes.  On 
the  other  hand,  digital  audio  contains  a 
discrete  range  of  frequencies  and/or 


amplitudes.  This  type  of  audio  is 
represented  by  stereo  system  compact 
disks  (CDs),  which  prt^uce  good  quality 
sound. 

Using  audio  with  CBT  requires  the  user 
or  system  to  synchronize  the  audio  to  the 
CBT.  If  the  user  is  responsible  for 
synchronizing  the  audio  to  the  CBT,  the 
user  must  interact  with  the  audio  system. 
Generally,  this  procedure  involves 
turning  on  and  off  an  analog  tape  player. 
The  system  can  also  turn  the  tape  player 
on  and  off  for  the  user,  which  may 
require  the  CBT  station  to  have  another 
piece  of  equipment.  If  the  CBT  is 
updated,  the  audio  tape  must  be  updated 
in  addition  to  updating  the  training 
software.  If  digital  audio  is  used,  it  can 
be  contained  on  the  CD-ROM,  thereby 
eliminating  the  need  for  an  external  tape 
player.  Additionally,  using  digital  audio 
on  a  CD-ROM  permits  random  access  to 
the  audio  segments,  a  feature  not 
available  in  a  linear  tape  mechanism.  If 
a  CD-ROM  is  used  when  the  CBT  is 
updated,  only  the  CD-ROM  will  be 
updated. 

Digital  Audio  Digital  audio  is  a  digital 
representation  of  the  analog  audio.  The 
conversion  of  analog  audio  to  digital 
audio  is  similar  to  the  process  used  in 
creating  CDs.  The  analog  audio  is 
sampled  at  specified  time  intervals  and 
converted  into  a  digital  representation. 
When  digital  audio  is  played,  the  digital 
representation  is  convert^  into  an 
analog  form,  which  is  then  sent  to  the 
speaker. 

The  quality  of  the  digital  audio  is 
determined  by  several  factors.  The 
output  sound  can  only  be  as  good  as  the 
sum  of  the  worst  elements  in  the 
production  of  the  audio.  The  factors 
affecting  the  quality  of  the  digital  audio 
sound  are  the  quality  of  the  original 
recording,  the  device  used  to  convert  the 
analog  audio  into  digital  audio,  the 
device  used  to  convert  the  digital  audio 
back  to  an  analog  form,  and  the  speaker 
used  to  produce  the  audio. 
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The  most  important  factor  affecting  the 
sound  quality  is  the  original  recording. 

If  the  origin^  recording  is  low  quality, 
the  rest  of  the  system  can  do  little  to 
improve  the  qu^ity  of  the  sound.  To 
improve  a  low-quality,  original 
recording  requires  audio  processing 
techniques,  which  are  beyond  the  scope 
of  this  paper.  Furthermore,  varying 
amplitudes  of  the  original  recoiding  can 
have  adverse  effects  on  the  digital  audio. 
However,  some  corrections  can  made 
during  the  digitizing  process,  but  it  is 
easier  to  record  at  a  constant  level. 

The  digitizing  device  affects  the 
conversion  of  analog  audio  into  a  digital 
audio.  The  state  of  digitizing  electronics 
is  so  good  that  this  step  introduces 
almost  no  distortion  into  the  digital 
audio.  The  sampling  rate  of  the  analog 
signal  can  have  a  dramatic  effect  on  the 
audio.  The  faster  the  sampling  rate,  the 
closer  the  digital  representation  is  to  the 
analog  signal.  As  the  sampling  rate 
increases,  the  amount  of  digital  data 
increases.  Sampling  rates  at  4, 6,  8,  12 
kbytes  were  available  for  the  digital 
recording  of  the  RPT  Trainer,  ^ch 
sampling  rate  allows  different  ranges  of 
sound  to  be  digitized.  The  limitations 
imposed  on  the  dynamic  sound  ranges 
are  tabulated  below: 


Sampling  Rate  Digitized 

per  Second  Audio 

Range 

4  kbytes  300  Hz  to 

2  kHz 

6  kbytes  300  Hz  to 

3  kHz 

8  kbytes  300  Hz  to 

4  kHz 

12  kbytes  300  Hz  to 

6  kHz 


lower  sampling  rates  produced 
unacceptable  sound  quality.  The  12- 
kbyte  sampling  rate  produced  high- 
quality  sound  but  was  deemed 
unnecessary  for  the  project  based  on  the 
sound  requirements  and  the  choice  of  a 
speaker. 

The  size  of  the  audio  file  produced  is 
dependent  on  the  sampling  rate.  The 
lower  sampling  rate  produced  smaller 
files.  Five  seconds  of  audio  sampled  at 
8  kbyte  creates  a  file  of  approximately 
44  kbytes.  Additional  Hie  space  is 
required  in  the  cataloging  process  of  the 
au^o.  A  crude  calculation  will  show 
that  at  an  8-kbyte  sampling  rate,  over  18 
hours  of  audio  can  be  stored  on  a  CD- 
ROM. 

Varying  output  levels  in  audio  were 
primarily  due  to  the  digitizing  process. 
Therefore,  care  should  be  exercised  to 
maintain  a  constant  level  of  input  audio 
when  digitizing  audio. 

When  the  audio  is  played,  the  digital 
representation  is  converted  to  analog 
form,  which  is  then  sent  to  the  speaker. 
The  state  of  digitizing  electronics  is  so 
good  that  this  step  pi^uces  almost  no 
detrimental  effect  on  digital  audio. 
However,  the  speaker  car*  have  a 
dramatic  effect  on  the  sound  produced. 

If  the  speaker  reproduces  the  sound 
poorly,  the  rest  of  the  system  can  do 
little  to  compensate  for  the  speaker.  The 
RPT  Trainer  used  a  headset  speaker, 
which  allowed  a  lower  sampling  rate. 

Limitations  Imposed  by  Digital  Audio 
The  primary  limitation  of  digital  audio  is 
that  it  is  a  "memory  hog."  The  higher 
the  sound  quality  of  the  audio,  the  more 
disc  space  is  required.  The  sound 
quality  desired  should  be  evaluated  at 
the  beginning  of  every  project,  and 
different  levels  of  sound  quality  can  be 
intermixed  in  a  CBT. 


We  found  that  a  sampling  rate  of  6  to  8 
kbytes  produced  sufficient  sound  quality 
for  RPT  Trainer  requirements,  whereas 


The  large  capacity  of  the  CD-ROM 
minimizes  the  impact  of  the  memory 
requirements  for  digital  audio.  Because 
of  the  access  speed,  we  discovered  that 
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short  audio  sequences  flowed  into  the 
program  much  better  than  did  the  long 
sequences.  The  attention  span  of  the 
learner  is  another  reason  to  keep  the 
audio  sequences  relatively  short.  The 
quality  of  the  digital  audio  was  noi 
^fected  by  the  ^-ROM,  and  there  was 
no  decrease  in  the  clarity  of  the  spee.h. 

File  Organization  Optimization  The 
slower  access  time  of  the  QD-ROM  was 
a  major  concern  in  the  development  of 
the  CBT  file  system.  The  CBT  design 
uses  a  large  number  of  small  external 
files,  and  we  were  concerned  about  the 
time  needed  to  open  and  close  these 
files.  However,  we  observed  very  little 
delay  on  the  PC  80386  machine.  We 
organized  the  files  on  the  CD-ROM  to 
optimize  the  access  time  by  creating 
separate  subdirectories  for  large  picture 
files,  audio  files,  data  files,  and 
executable  code  ^hus  reducing  the  seek 
time  to  the  large  files.  This  optimization 
was  easily  accomplished. 

CD-ROM  Simulator  As  a  technical 
note,  we  used  an  Advanced  Graphic 
Applications  (AGA)  magneto-optical 
disc  drive  to  simulate  a  CD-ROM.  The 
AGA  drive  uses  a  two-sided  cartridge 
with  320  Mbytes  on  each  side.  To 
access  data  on  the  other  side  of  the 
optical  disc  requires  removing  the  disc 
and  flipping  it  over.  The  use  of  the 
AGA  dnve  imposed  some  limitations 
during  the  development  process.  One 
limitation,  however,  was  the  number  of 
allowable  DOS  buffers.  The  AGA 
software  requires  a  patch  DOS  to 
version  3.3,  which  increases  the  size  of 
each  buffer  from  512  bytes  to  8192 
bytes.  If  the  program  requires  a  large 
number  of  buffers,  there  may  not  be 
enough  memory  left  to  load  and  execute 
the  program.  Because  the  RPT  Trainer 
did  not  need  many  buffers,  we  were  able 
to  reduce  the  number  of  buffers  in  the 
config.sys  file  to  five. 

Conclusions  CD-ROM  is  a  viable 
alternative  for  multimedia  CBT  delivery. 
Its  large  capacity  allows  the  user  to 
incorporate  scanned  images,  large 


computer-generated  graphics,  and  digital 
audio  in  a  large  CBT  program.  Once  set 
up,  the  development  for  the  CD-ROM  is 
nearly  identic^  to  the  development 
process  for  traditional  delivery  systems. 
Because  the  technical  aspects  of  CBT 
development  for  CD-ROM  are  trivial, 
anyone  who  produces  CBT  courseware 
can  deliver  it  on  a  CD-ROM,  which 
enables  CBT  developers  to  maximize  the 
time  spent  on  courseware  development. 
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ABSTRACT 

This  paper  focuses  on  the  changing  direction  within  the  Air  Training  Command  (ATC)  Systems  Support  Activity 
(SSA)  to  meet  the  growing  need  for  multiple  software  tools  to  support  Interactive-courseware  (ICW)  develop¬ 
ment.  In  the  past,  the  SSA’s  main  function  was  to  produce  the  Merlin  authoring  system.  Now,  with  the  emphasis 
on  accomplishing  more  training  “at  the  job  site”,  more  rapid  course  production  is  necessary.  This  requires 
shorter  timelines,  use  of  subject  matter  experts  with  little  or  no  computer  experience  and  the  fielding  of  new 
courses  quickly.  This  evolution  drove  ATC  to  rethink  its  philosophy  regarding  software  tool  development. 

The  purpose  of  this  paper  is  two-fold: 

1)  detail  the  change  in  direction  being  taken  by  ATC  and  the  SSA. 

2)  provide  a  description  of  tools;  already  produced,  under  development, 
or  in  the  planning  stage. 

This  paper  is  aimed  at  ICW  developers  at  all  levels  of  expertise.  In  fact,  it  was  written  m  response  to  concerns 
voiced  by  subject  matter  experts  with  little  knowledge  of  computer  programming  or  software  development  who 
have  been  charged  with  producing  ICW. 


BIOGRAPHY 

Cupt  Bill  Coffey  -  is  Chief,  Customer  Support  Branch,  3302d  Technical  Training  Squadron  (ATC  Systems 
Support  Activity),  Keesler  AFB  .MS.  His  previous  assignment  was  as  Chief,  Training  Technology  Plans  Branch, 
34(X)  Technical  Training  Wing  (ATC),  Lowry  AFB  CO.  Captain  Coffey  holds  a  BS  in  Education  from  Oklahoma 
City  University  and  an  MA  in  .Management  Information  Systems  from  Webster  University.  He  has  produced 
several  ICW  courses  using  the  Merlin,  Tcncore,  and  Quest  authoring  tools  and  has  helped  design  many  of  the 
tools  we  are  now  developing. 


180 


CREATING  SOFTWARE  TOOLS 
FOR  ICW  DEVELOPERS 

(A  SYSTEMS  APPROACH) 


INTRODUCTION 

The  production  of  high  quality  Interactive- 
courseware  or  ICW  is  often  a  time  consuming  and 
costly  effort.  For  example;  the  learning  curve  for 
development  of  graphics  and  adequate  testing  strate¬ 
gics  is  often  long  and  complex.  It  takes  time  for  new 
developers  to  become  proficient  with  the  selected 
authoring  tool.  In  addition,  there  are  up  front  costs 
associated  with  purchasing  the  complete  compliment 
of  software  required  to  do  the  job.  Our  goal  is  to 
provide  novice  developers  with  low  cost,  easy  to  use 
tools  that  reduce  up  front  costs  and  shorten  both  the 
learning  curve  and  development  time. 

One  way  to  accomplish  our  goal  -  to  give  our 
developers  the  very  best  tools  possible  -  is  by  taking 
a  systems  approach  towards  development.  That  is, 
we  must  field  a  complimentary  suite  of  software  prod¬ 
ucts  or,  in  other  words  a  “software  toolbox”,  that 
integrates  alt  phases  of  ICW  development  into  an  easy 
to  use  package.  Such  tools  provide  the  developer  with 
“canned  routines”  or  “preformatted  shells”  that  only 
require  the  user  to  “fill  in  the  blanks”  to  get  the  job 
done.  By  eliminating  the  need  to  manually  produce 
lesson  code,  or  use  the  “pixel  by  pixel”  method  for 
creating  graphics,  we  enable  the  developers  to  con¬ 
centrate  on  their  most  important  task  -  educationally 
sound  lesson  design.  We  must  also  provide  the  nec¬ 
essary  training  on  how  to  use  these  tools.  Ideally,  this 
training  should  be  computer-based  and  accompany 
the  products  that  we  distribute. 

In  recognition  of  this  fact.  Air  Training  Command 
through  its  System.'  Support  Activity  (SSA)  located  at 
Keeslcr  AFB  in  Mississippi,  is  shifting  its  locus  away 
from  the  production  of  a  single  authoring  tool  - 
MERLIN  -  in  favor  of  developing  several  compli¬ 
mentary  software  tools  to  aid  ICW  developers.  The 
SSA  was  selected  for  this  role  because  it  is  the  ATC 
focal  point  for  all  training  related  system  development 
and  life  cycle  support.  Further,  the  SSA  is  undergr)ing 
a  restructuring  of  its  staff  to  balance  the  number  of 
technical  staff  members  with  the  number  of  training 
specialists.  This  is  necessary  to  ensure  that  the  tools 
developed  by  the  SSA  arc  geared  toward  the  main 
target  audience  within  the  Air  Force  -  subject  matter 
experts  with  little  or  no  cx)mputer  programming  expe¬ 
rience. 


In  later  paragraphs  each  of  the  following  softw'are 

tools  will  be  discussed  in  greater  detail: 

A.  GRAPHICS: 

A  graphics  database  program  containing  graph¬ 
ics  available  for  use  to  decrease  the  amount  of  time 
a  developer  must  spend  in  creating  new  graphics. 

B.  TEST  ADMINISTRATOR; 

A  program  to  create,  administer,  and  score  tests. 
This  program  can  also  analyze  test  results  when 
coupled  with  the  Test  Analysis  or  TAP  package. 

C.  KEYBOARD  TRAINER; 

A  basic  typmg  tutorial  aimed  at  non-proficient 
typists  to  help  improve  keyboard  familiarity,  speed 
and  accuracy. 

D.  ARTS: 

An  Adaptive,  Randomized  Test  Sequencer  that 
provides  a  step-by-step  format  for  writing  objectives 
and  test  questions. 

E.  ACCD: 

The  Automated  Course  Con'rol  Documentation 
(ACCD)  system  is  being  developed  to  enable  course 
developers  (both  ICW  and  non-ICW)  to  produce  & 
update  course  documents. 

F.  MERLIN  32: 

This  is  the  non-IVD  version  of  the  Merlin  Author¬ 
ing  tool.  It  is  designed  to  run  on  a  standard  MS- 
DOS  PC  with  640k  of  RAM. 

G.  MERLIN  33: 

An  Interactive-video  version  of  the  MERLIN 
authoring  tool.  Specifically  aimed  at  course  devel¬ 
opers  who  use  the  DVA-4000/1SA  Board  as  their 
presentation  media. 

H.  MERLIN  FRONT-END: 

A  Graphical  User  Interface  which  negates  the 
need  for  developers  to  write  Merlin  code.  It  offers 
pull  down  menus  that  enable  a  user  to  easily  create 
a  lesson. 

BACKGROUND 

According  to  the  new  Department  of  Defen.se  In¬ 
struction  (1322.20),  for  the  Development  and  Man¬ 
agement  of  ICW,  dated  March  14,  1991;  “The 
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decision  to  use  ICW  shall  be  based  on  a  comprehens¬ 
ive  analysis  of  the  total  training  system  requirements, 
and  a  media  selection  analysis  to  determine  if  the  use 
of  ICW  is  an  effective  and  efficient  means  for  present¬ 
ing  training  materials  when  compared  with  other  po¬ 
tential  training  media.” 

There  are  many  promising  programs  which  cannot 
make  it  past  this  evaluation  process  due  to  cost  and 
time  constraints.  Many  of  these  programs  could  ben¬ 
efit  training  and  in  the  long  term  prove  to  be  very  cost 
effective.  However,  they  rely  on  costly  methods  for 
development  (due  to  a  lack  of  adequate  software 
tools),  and  must  u.se  off-the-shelf  authoring  packages 
which  often  price  the  program  out  of  the  range  where 
it  is  cost  effective. 

In  response  to  these  user  needs,  the  ATC  SSA  is 
now  focusing  on  more  “up-iVont”  user  involvement  in 
developing  ICW  software  tools.  This  means,  that  our 
users  will  have  a  greater  voice  in  what  tools  we  de¬ 
velop,  how  they  are  designed,  and  what  changes  in 
configuration  will  be  allowed.  By  providing  the  user 
with  more  input  during  the  development  process,  we 
can  produce  integrated  tools  in  a  more  timely  and  cost 
effective  way.  The  SSA  has  the  organic  capability  to 
produce  useful  software  tools  which  can  be  used  by 
ICW  developers  to  compress  lengthy  timelines 
thereby  reducing  development  costs.  When  the  time 
it  takes  to  generate  good  ICW  can  be  reduced,  pre¬ 
viously  unsupported  projects  gain  rapid  approval  for 
implementation.  The  end  result  is  a  tangible  training 
benefit  to  the  DoD. 

The  list  of  tools  presented  earlier  represents  the 
SSA’s  response  to  requirements  presented  to  us  by 
our  customers;  the  ATC  Training  Centers,  Air  Force 
Major  Commands,  the  Air  National  Cuard  and  Re¬ 
serve,  and  the  other  services.  It  is  not  a  complete  nor 
conclusive  list  but,  rather  an  initial  reply  to  the  need 
for  more  efficient  tools  to  aid  ICW  development. 
What  our  users  want  is  more  training  related 
functionality  rather  than  more  technical  capabilities. 

What  users  need  to  do  their  job  effectively  arc  soft¬ 
ware  tools  developed  in  response  to  training  require¬ 
ments  and  not  software  packages  that  force  users  to 
change  their  instructional  design  to  match  the  design 
of  the  available  software. 


RESPONDING  TO  THE  NEED 

To  meet  the  growing  demand  for  easy  to  use,  inte-  | 
grated  software,  the  SSA  is  developing  or  maintaining  j 
the  following  tools:  I 

A.  GRAPHICS  ! 

One  of  the  most  time  consuming  tasks  in  ICW  | 
development  is  the  production  of  graphics.  In  many  ■ 


cases  the  ICW  organization  does  not  have  the  talents 
of  a  graphics  specialist  available  and  this  job  falls  to 
the  “most  qualified”  person  on  the  team.  Normally, 
this  results  in  less  than  professional  products  (which 
take  an  inordinate  amount  of  time  to  produce) 
being  used  in  the  lessons.  Several  surveys  were  ac¬ 
complished  but,  no  government  or  off-the-shelf 
commercial  package  could  be  found  that  met  the 
needs  of  our  customers. 

A  graphics  database  program  (written  in  ’C’  and 
now  being  re-written  in  ADA)  has  been  developed 
to  decrease  the  amount  of  time  a  developer  must 
spend  in  creating  new  graphics.  This  database  al¬ 
ready  contains  over  2000  PCX  format  graphics  and 
is  growing  every  day.  It  operates  using  a  graphical 
interface  and  is  very  user  friendly.  The  program 
(figure  1)  allows  a  user  to  retrieve  graphics  by 
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searching  the  entire  database  or,  by  the  user  enter¬ 
ing  from  1  to  4  keywords  to  narrow  the  search 
pattern. 

The  database  is  alphabetized  and  can  store  as 
many  graphics  as  there  is  system  storage  space.  The 
user  may  elect  to;  view  the  graphics  on  the  screen, 
copy  the  graphics  to  floppy  disk,  or  copy  the  graph¬ 
ics  to  another  directory  location  on  the  same  com¬ 
puter  containing  the  graphics  database. 
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B.  TEST  ADMINISTRATOR 

In  response  to  a  request  from  the  USAF  Person¬ 
nel  and  Information  Management  Specialist 
Schools,  the  SSA  created  a  program  that  enables  a 
subject  matter  specialist  to  create,  administer,  and 
score  tests.  The  program  was  written  to  allow  in¬ 
structors  (even  with  no  knowledge  of  the  test  file 
structure)  to  develop  a  test.  Through  the  use  of  a 
simple  menu  system  (figure  2),  the  instructor  can 
develop  a  question  pool  and  then  administer  and 
score  the  test. 

C.  KEYBOARD  TRAINER: 

Recognizing  the  fact  that  different  people  oper¬ 
ate  a  keyboard,  at  various  levels  of  proficiency,  the 
SSA  developed  a  basic  typing  tutorial  to  help  im¬ 
prove  keyboard  familiarity,  speed  and  accuracy. 
This  product  is  a  generic  tool  that  has  application 
for  both  beginner  and  advanced  typist  alike. 

D.  ARTS: 

Randomized,  adaptive  testing  refers  to  a  rela¬ 
tively  new  instructional  technology  that  is  designed 
to  decrease  measurement  error,  while  keeping  test 
taking  time  to  a  minimum.  ARTS  was  developed  at 
Lowry  Technical  Training  Center  to  meet  the  need 
for  an  adaptive  and  randomized  test  development 
and  delivery  system.  The  Adaptive,  Randomized 
Test  Sequencer  provides  a  step-by-step  tutorial  on 
how  to  write  objectives  and  test  questions.  It  also 
provides  for  a  “test  pool”  of  questions  that  randomly 
selects  questions  to  be  presented  at  the  time  the  test 
is  taken. 

ARTS  consists  of  three  integrated  modules: 

Module  1  -  The  ARTS  Generator  aids  the  devel¬ 
oper  in  creating  the  test. 

Module  2  -  The  ARTS  Presentation  randomly  se¬ 
lects  and  presents  the  test. 

Module  3  -  The  ARTS  Report  Generator  compiles 
a  record  of  student  testing  used  by  an  instructor 
to  obtain  a  breakdown  of  student  performance. 

ARTS  Features: 

The  system  can  create  a  test  pool  which  contains 
a  maximum  of  10  objectives  with  15  possible  ques¬ 
tions  for  each  objective  or  150  total  questions.  Each 
question  contains  space  for  the  correct  answer  and 
up  to  4  distractors.  The  program  can  generate  ei¬ 
ther  multiple  choice  or  true/false  questions.  A  fu¬ 
ture  version  of  ARTS  will  include  the  capability  for 
matching  and  short  answer.  The  randomization  fea¬ 
ture  ensures  that  each  student  will  be  presented  a 
different  version  of  the  test  while  making  sure  that 
all  necessary  objectives  are  tested. 


(The  minimum  system  configuration  required  to 
run  ARTS  is  shown  in  figure  3.)  ARTS  uses  four 
main  menus  or  screens  to  “walk  the  user”  through 


SYSTEM  REQUIREMENTS 

MS-DOS  computer.... 

....IBM  ST/AT  or  compatible 

Minimum  Memory . 

....640k  required 

Disk  Drives . 

. Roppy  Disk  drive 

1  (Load  and  run  ARTS  from  a  hard  disk  drive  if  desired) 

Monitor . 

. Any  model  will  suffice 

MS-DOS . 

. Version  2.0  or  later 

Figures  -  arts  SYSTEM  requirements 
the  testing  cycle.  Each  of  these  screens  are  shown 
below  as  figures  4  through  7 


ADAPTIVE  AND  RANDOMIZED  TEST  SEQUENCER  (ARTS) 

TEST:  cdissl 

OPR:  lenry  1  penna 

DATE  CREATED:  8  JAN  91 
LAST  DATE  MODIFIED:  10  JAN  91 
COMMENTS: 

I  cam*  iniroducuon  kason  pc  ] 


NUMBER  OF  OBJECTIVES  FOR  THIS  L3SSON?  1  1  ) 


FI  •  Help  Screen 

F3  •  Load  New  Lesson  File 

F5  •  Delete  Current  Screen 

F2  •  Save  to  File 

F4  -  Restore  Current  Screen 

FIO  •  Exit  Program 

Figure  4  - 

ARTS  INITIAL  DATA 

ADAPTIVE  AND  RANDOMIZED  TEST  SEQUENCER  (ARTS) 

-•  LESSON  OBJECTIVE  #2 


Number  o(  queauons  (o  be  prcaenied?  (  IS  ] 

Masmum  number  of  questions  (o  be  presented?  (  10  ] 

Minimum  percentage  required  for  passing  this  objective? 

FI  •  Help  Screen  F3  •  Load  New  Lesson  File  F5  -  Dcleic  Current  Screen 

F2  •  Save  to  Fite  F4  -  Restore  Current  Screen  FIO  •  Exit  Program 

Figure  5  ■  arts  lesson  objectives 

Another  important  feature  of  ARTS  is  that  it  can 
be  used  with  any  Authoring  System  or  Language 
which  has  the  capability  to  “call  external  programs” 
such  as  DOS  executables.  This  allows  the  developer 
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Student  Name:  GRAY 

Student  failed  to  pau  one  or  more  objectives 

Objective  number  1 

Identify  facts,  pnneiplea,  and  relationships  associated  with  titc  Adaptive  and 
Randomized  Test  Sequence  system. 

Student  failed  this  objective 

Question  Student 

Correct 

Presented  Response 

Answer 

1  A 

B 

2  D 

A 

Figure  7  -  arts  test  summary 


ADAPTIVE  AND  RANDOMIZED  TEST  SEQUENCER  (ARTS) 

SUMMARY  - 

TEST  NAME:  cams 

OBJECTIVE  NUMBEROF 
QUEynONS 

1  15 

MAXIMUM  ^cTO 

PRESEN''rED  PASS 

10  70 

NUMBER  OF  OBJECTIVES;  1 

OPR:  terry  1  penna 

DATE  CREATED'  8  JAN  91 

LAST  DATE  MODIFIED:  lOJANvi 

Press  RETURN  key  to  conunuc 

Figure  6  -  student  summary 
to  use  ARTS  across  a  broad  spectrum  of  media  and 
keeps  it  independent  from  the  authoring  tool. 

E.  ACCD: 

The  Automated  Course  Control  Documentation 
(ACCD)  system  is  being  developed  to  help  course 
developers  (ICW  and  non-ICW)  to  generate  and 
quickly  update  the  various  documents  used  within  a 
course.  The  program  is  being  written  to  “issue  Hags  ’ 
which  alert  the  trainer  that  there  are  multiple  docu¬ 
ments  affected  each  time  they  make  a  change  to  one 
of  a  series  of  related  course  documents. 

F.  MERLIN  3 

This  version  of  Merlin  is  almost  identical  to  version 
3.3,  except  it  does  not  contain  the  IVD  capabilities. 

G.  MERLIN  3 J; 

An  Interactive-video  version  of  the  MERLIN  au¬ 
thoring  tool.  Specifically  aimed  at  course  developers 
who  use  the  DVA-4000/ISA  Board  to  drive  their 
presentation  media. 
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H.  MERLIN  FRONT-END: 

This  program  will  enable  a  user  to  create  Merlin 
lessons  using  a  series  of  “preformatted  templates” 
which  negates  the  need  to  actually  use  Merlin  code. 
It  offers  pull  down  menus  that  enable  a  user  to  easily 
create  a  lesson  by  following  the  menu  structure  (sim¬ 
ilar  to  a  standard  DOS  tree  structure)  for  each  specific 
area.  The  menu  system  will  allow  a  user  to  develop 
initial  screen  defaults  for  text,  graphic,  question,  and 
branching  functions.  Once  a  user  creates  a  default 
profile  (say  all  positive  question  feedback  is  white  text 
on  cyan  background)  this  default  is  carried  through  to 
all  screens  developed  for  questions  and  answers. 

CONCLUSION 

Within  the  Department  of  Defense,  the  pool  of 
resources  available  for  developing  and  conducting 
required  training  will  most  likely  continue  to  shrink. 
There  will  be  an  ever  increasing  need  to  find  new  and 
more  cost  effective  ways  for  meeting  our  training 
requirements.  Also,  it  is  quite  evident  that  more  train¬ 
ing  will  be  conducted  at  the  user  location  using  some 
form  of  exportable  courseware.  In  general,  this 
courseware  will  be  ICW. 

Subject  Matter  Specialiststs  (SMSs)  will  continue 
to  be  the  primary  developers  of  ICW  within  the  Air 
Force.  They  will  require  more  sophisticated  and  per¬ 
haps  more  ii.ielligent  tools  to  aid  them  in  their  devel¬ 
opment  efforts.  It  is  not  our  aim  to  make  “computer 
programming  experts”  of  these  SMSs.  Our  goal  is  to 
provide  them  with  tools  to  automate  functions  so  that 
the  SMS  can  concentrate  on  the  design  and  content  of 
a  course  and  not  spend  countless  hours  worrying 
about  the  technical  details  of  production. 

The  ATC  SSA  will  continue  developing  more  soft¬ 
ware  tools  to  help  SMSs  do  their  jobs.  Additionally, 
we  see  our  role  evolving  to  encompass  more  training 
of  users,  more  analysis  of  user  requirements,  and 
more  hands-on  development  of  ICW  used  to  train 
people  on  how  to  use  our  products.  The  keys  to 
continued  success  are;  user  involvement  in  all 
phases  of  design  and  development,  quality  product 
support  which  includes  training,  and  maintaining 
the  systems  approach  to  our  efforts! 


ENDNOTES 

1.  ARTS  and  GRAPHICS  were  developed  by  ILts 
Doug  Rausch  and  Steve  Gray,  3400  Technical 
Training  Wing  (ATC),  Lowry  AFB,  CO. 

2.  The  DVA-4000/ISA  Board  is  a  proprietary  piece 
of  hardware  produced  by  Video  Logic  Limited. 
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ABSTOMTT 

This  paper  describes  the  design  and  iirplementation  of  an  automated  data 
interface  supporting  the  concurrent  Logistic  Support  Analysis  (I£A)  and 
Instruc^ioncil  Systems  Development  (ISD)  processes.  The  Depsartment  of  Defense 
(DoD)  Caiputer-Aided  Acquisition  and  Logistic  Support  (CALS)  initiative 
requires  the  development  of  standard  weapon  system  data  bases  that  can  be  used 
to  support  front-end  logistics,  training,  and  p)erformance  analysis  of  new  or 
emerging  weapon  systems.  CALS  includes  the  evolving  concept  of  an  Integrated 
Weapon  System  Data  Base  ( IWSDB ) ,  a  logical  ( as  opposed  to  physical )  collection 
of  all  data  related  to  a  weapon  system's  acquisition,  design,  cperation,  and 
support.  Training  analyses,  as  well  as  other  LSA  ccnponent  processes,  would 
have  automated  access  to  current  weapon  system  data  that  are  pertinent  to 
training  system  development  decision  making.  The  CALS  IWSDB  requires  standard 
definitions  of  data  elements  and  data  exchange  standards  that  overcome 
inherent  hardware/software  inccxipatibilities .  The  ISD-LSA  interface  described 
in  this  p)ap)er  deponds  upon  standaurd  definitions  of  training  data  elements  and 
unique  data  exchange  stcindards  and  procedures.  The  ISD-LSA  CALS  (Phase  1) 
interface  described  in  this  papor  is  acccnplished  through  data  exchange 
between  an  ISD  analysis  tool  and  the  LSA  Record  (LSAR)  within  the  CALS  IWSDB. 
The  structure  of  IWSDB  (ISAR)  input  and  output  interchange  data  files  are 
described.  In  addition  to  the  data  elements  and  relationships  comprising  the 
files,  data  control  issues  cure  addressed. 
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The  Joint  Service  ISDA-^^R  nfiS 

The  Joint  Service  Instmctional 
Systems  Develcpnent/Logistic  Si^port 
Analysis  Record  (ISD/ISAR)  Etecision 
Support  System  (DSS)  is  a  major 
Department  of  Defense  (DoD)  effort  to 
better  si^jport  ISD  decision  naking  and 
to  integrate  trciining  system 
development  with  other  weapon  system 
design  activities.  The  PC-beised 
multi-user  system  consists  of  data 
input,  ISD  analysis,  and  trciining 
system  design  procedures  that  reflect 
and  accatmodate  service-specific  ISD 
methodologies . 

The  key  feature  of  the  Joint  Service 
ISD/LSAR  DSS  is  the  automated  I£AR-to- 
ISD  data  interface.  The  interface 
permits  a  training  system  develc^ment 
to  maintain  concurrency  with  the 
evolving  weapon  system  design  and 
SD^jpxartability  characteristics 
recorded  in  the  ISAR.  The  decisicxi 
si^pport  techniques  esiployed  by  the  DSS 
inprove  and  standardize  ISD  decision 
making  by  providing  users  with 
c^spropriate  cuid  ccHisistent 
presentaticxis  of  I£AR  and  other 
training-related  data. 

The  DSS  COTisists  of  IfiAR  data  ir^t 
routines  and  Joint  Service  ISD 
analysis  pirocesses.  The  system 
includes  utility  functions  that 
provide  system  security,  database 
administration,  report  generation,  and 
ISD  anailysis  functicxis.  The  DSS 
autcmated  procedures  are  presented  in 
Figure  1  and  consist  of  the  following ; 

•  ISD/LSAR  Data  Interface  Functions 

•  Administrative  Functions  including 

data  security 

•  ISD  Program  Management  Ehnctions 

•  ISD  Analysis  Procedures 

•  Select  Teisks  for  Training,  using 

the  following  models; 


Sub-Task  Analysis  Model  (SIAM)  for 
Task  Selection 
8-Factor  Model 
4-Factor  Model 
Difficulty,  Inportance,  and 
Frequency  (DIF)  Model 
Early  Conparability  Analysis  (EGA) 
Model 

•  Select  Instructional  Settings 

•  Develop  Teaming  Cfcjectives 

•  Determine  Instructional  Sequence/ 

Develop  Course  Structures 

•  Seleort:  Training  Media,  using  the 

following  models: 

Sub-Task  Analysis  Model  (STAM) 
for  Media  Selection 
Automated  Instructional  Media 
Selection  (AIMS)  Mooiel 

•  Identify  Fidelity  Requirements  of 

Training  Devices 

•  Identify  Training  Devic:e 

Instructional  Features 

The  Joint  Service  ISD/LSAR  DSS  LSAR 
interface  ijiproves  the  quality  of 
infcsrmaticn  exchange  between  ISD 
anedysts  and  system/equipment 
designers,  providing  the  ability  to 
address  a  wider  range  of  training 
issues  in  a  more  ccnplete  fashion. 

Both  the  early  analysis  of  training 
requirements  and  the  systems 
engineering  interaction  with  the 
equipment  designers  contribute  to  the 
development  of  more  effective  training 
systems. 

LSAR  Data  Preaentatiops 

One  of  the  unique  features  of  the 
Joint  Service  ISD/ISAR  DSS  is  its 
interfac:e  with  LSAR  data.  Meaningful 
presentations  of  ISAR  training-related 
oiata  support  the  analyst  in  perf  oirming 
ISD  analyses  and  in  making  effeortd.ve 
ISD  decisions.  The  DSS  uses  ISAR  data 
in  one  of  two  ways.  First,  LSAR  data 
that  descnribe  a  weapon  system's 
equipment  strvicture,  task  hierarchy. 
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Table  I  LSAR  Data  Elements  in  the  DSS  Data  Set 


May  1990  Draft  MIL-STD-1388-28  HIL-STD-1388-2A 


Data  Element  Title 

DED 

DED 

ALTERNATE  LCN  CODE 

019 

023 

ANNUAL  NUMBER  OF  MISSIONS 

021 

027 

ANNUAL  OPERATING  DAYS 

022 

028 

ANNUAL  OPERATING  REQUIREMENTS 

023 

029 

END  ITEM  ACRONYM  CODE 

093 

106 

HARDNESS  CRITICAL  PROCEDURES  (HCP) 

148 

153 

HAZARDOUS  MAINTENANCE  PROCEDURES  CODE 

151 

155 

LCN  NOMENCLATURE 

195 

181 

LSA  CONTROL  NUMBER  (LCN) 

193 

197 

MEAN  EUPSED  TIME 

217 

220 

MEAN  MAN-MINUTES 

219 

223 

MEAN  MINUTE  ELAPSED  TIME 

220 

232 

MEAN  MISSION  DURATION 

221 

234 

MEANS  OF  DETECTION 

230 

242 

MEASUREMENT  BASE  (MB) 

231 

244 

OPLfiATIONAL  REQUIREMENT  INDICATOR 

268 

285 

PERFORMANCE  STANDARDS 

280 

313 

FACILITIES  TRAINING  REQUIREMENT  CODE 

350A 

394A 

TRAINING  EQUIPMENT  REQUIREMENT  CODE 

350B 

394B 

TOOL/SUPPORT  EQUIPMENT  REQUIREMENTS  CO 

350C 

394C 

SEQUENTIAL  SUBTASK  DESCRIPTION 

364 

410 

SKILL  LEVEL  CODE 

378 

422 

SKILL  SPECIALTY  CODE  (SSC) 

379 

423 

SKILL  SPECIALTY  EVALUATION  CODE 

380 

433 

SUBTASK  NUMBER 

399 

451 

TASK  CODE 

419 

467 

TASK  CONDITION 

420 

468 

TASK  CRITICALITY 

421 

469 

TASK  FREQUENCY 

422 

470 

TASK  IDENTIFICATION 

423 

472 

TECHNICAL  MANUAL  CODE  (TM  CODE) 

429 

479 

TRAINING  LOCATION  RATIONALE 

456 

502 

TRAINING  RATIONALE 

457 

503 

TRAINING  RECOMMENDATION 

458 

504 

WORK  AREA  CODE 

508 

544 

and  task  perfonrance  requirements 
prcjvide  the  ISD  analysis  structxire 
used  by  the  DSS.  The  ISD  cinalyst  uses 
LSAR  data  to  construc±  the  DSS 
subsystem,  task,  task  element, 
skill/kncwledge  hierarchy.  The  second 
purpose  of  ISIAR  data  is  to  support 
specific  ISD  decisions. 

The  Joint  Service  ISD/ISAR  DSS, 

Version  4.0  is  caipatible  vdth  data 
element  definitions  and  relationships 
in  MIIr-STD-1388-2A,  Logistic  Si^port 
Analysis  Record,  20  July  1984.  It  is 
desixeable  to  have  the  DSS  eventaial ly 
conform  to  the  emerging  revision  to 
the  LSAR  standard,  MIL-STD-1388-2B. 
Both  MTL-STDs  prescribe  the  data 
element  definitions,  data  field 
lengths,  and  data  entry  requirements 
for  I£AR  data.  MIL-STD-1388-2A  uses 
tMo  LSAR  naster  files;  the  ISA 
Control  Number  (LCN)  Master  File  and 
the  Task  Neurrative  Master  File.  The 
May  1990  draft  MIL-STD-1388-2B  uses  a 
relational  table  structure  of 
functional  data  element  groupings, 
vhere  MIL-STD-1388-2A  consists  of  card 
inages.  Even  though  the  new  standard 
continues  to  evolve,  most  inported 
training  data  elements  from  -2A  have 
been  preserved  in  the  -2B  version  with 
minor  modifications.  Table  1  provides 
a  cross-reference  of  the  training- 
related  ISAR  data  elements  fron  the 
two  MEL-STDs  that  support  DSS  trciining 
analyses.  The  data  elements  cure 
listed  by  both  MIL-3rD-1388-2A  eind 
MIL-STD-1388-2B  (May  1990  draft)  data 
element  definition  (DED)  nuirber. 

The  Joint  Service  TSn/TSAR  DSS  and 
CAES 

The  Department  of  Defense  (DoD) 
Ccnputer-Aided  Acquisition  eind 
Logistic  Support  (CAES)  initiative 
requires  the  develcpment  of  standard 
weapon  system  data  bases  that  can 
support  front-end  logistics,  training, 
and  performance  analysis  of  new  or 


emerging  we^xm  systems.  CAES 
includes  the  evolving  concept  of  an 
Integrated  Weapcm  System  Data  Beise 
(IWSDB),  a  logical  (as  opposed  to 
physiccd)  collection  of  cdl  data 
related  to  a  Training  analyses,  as 
well  as  other  ISA  ccnponent  processes, 
will  have  autcmated  access  to  current 
weapon  system  data  pertinent  to 
training  system  development  decision 
naking. 

New  we^»n  system  acquisition  programs 
are  being  required  to  be  "CAES 
cdipatible."  Both  industry  and 
Government  participants  in  nany  new 
system  acquisitions  are  attempting  to 
address  CALS  technical  issues  and 
realize  the  connunication  efficiencies 
and  cost  reductions  expected  from  CAES 
ccnpatibility.  The  effective  use  and 
exchange  of  we^x:>n  system  design  data 
in  a  CALS  environment  requires 
standard  definitions  of  data  elements 
and  data  exchange  standards  that 
overcome  inherent  inconpatibilities 
between  host  hardware/software.  The 
Joint  Service  ISD/ISAR  DSS  CAES 
interface  depends  upon  standard 
definitions  of  trcdning  data  elements 
and  unique  data  exchange  standards  and 
procedures. 

Operational  Overview 

Figure  2  presents  an  overview  of  the 
Joint  Service  ISD/ISAR  DSS  CAIS  (Phase 
1)  cCTipatible  autaiated  LSAR-to-ISD 
data  interface.  The  interface 
consists  of  several  major  data  stores 
and  data  processes.  The  data  stores 
depicted  in  Figure  2  reside  either 
within  the  CAIS  IWSDB  or  the  DSS  tool. 
Each  data  store  is  described  below; 

Within  the  IWSDB; 

ISAR  Data  -  System/equipment  logistics 
data  naintcdned  in  accordance  with 
MIL-STD-1388-2B. 
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ISD/DSS  CALS  Interface  Concept 


ISP  Data  -  A  logical  store 
(corresponding  to  one  or  more  physical 
stores)  of  training  system-related 
data.  The  data  nay  or  may  not  clLso 
reside  in  the  LSAR. 

Input  Interchange  Files  -  The  subset 
of  LSAR  data  that  is  used  to  perform 
ISD  analyses  vdthin  the  DSS. 

Output  Interchange  Files  -  Data 
resulting  frcm  ISD  analyses  performed 
using  the  DSS.  E>ata  in  this  file 
include  ISAR  data  elements,  as  well  as 
data  that  can  be  used  in  other 
concurrent  logistics  and  engineering 
processes. 

Within  the  Joint  Service  ISD/LSAR  DSS: 

Mirrored  Interchange  Files  -  The 
ccnplete  or  partial  IWSDB  interchange 
data  files.  The  mirrored  input  files 
contain  data  awaiting  analysis  within 
the  DSS;  the  mirrored  output  files 
contain  data  awaiting  transmission  to 
the  IWSDB. 

DSS  ISAR  and  ISD  Data  -  In-process 
LSAR  and  ISD  data. 

In  addition  to  the  above  data  stores. 
Figure  2  depicts  three  DSS  CAI£ 
interface  data  processes,  each 
described  belcw; 

Produce  Input  Interchange 
Files /Extract  Data  frcm  Output 
Interchange  Files  -  This  is  actually 
two  data  processes  represented  by  the 
"Interchange  File  Ifeintainer"  box  in 
Figure  2.  Creating  the  input 
interchange  files  requires  the 
performance  of  I£AR  data  ccmparisons 
(detecting  I£AR  updates /changes ) . 

Transfer  Interchange  Files  - 
Represented  in  Figure  2  by  the  "STD 
CCMM  PKG"  box,  the  transfer  of 
interchange  files  between  the  IWSDB 
and  the  DSS  is  acccnplished  using  the 


^prcpriate  host-to-PC  ccmnanications 
package  (not  part  of  the  DSS). 

Perform  ISD  Analyses /Produce  Output 
Interchange  Files  -  Performing  ISD 
analyses  using  the  DSS  is  unchanged  by 
the  CALS  ISAR  data  interface.  An 
additional  output  process  new 
constructs  the  output  interchange 
files  frcm  DSS  analysis  results. 

CATS  Tnterfaoe  Data  Flow  Process 
Descriptions 

Figure  3  displays  the  ISD/ISAR  DSS 
CALS  interface  in  the  form  of  a  data 
flow  diagram  (DFD) .  The  processes  and 
data  stores  presented  in  Figure  3  and 
described  belcw  expand  upon  those 
discussed  in  the  Operational  Overview 
and  displayed  in  Figure  2.  Ilie 
process  descriptions  indicate  the 
users /activities  that  cu:e  responsible 
for  process  develcpnent /performance. 
Many  processes  can  be  performed  by  a 
variety  of  users /activities.  Ihe 
user /activity  categories  "LSAR  owners" 
and  "ISD  data  owners"  indicate  those 
activities  that  exercise  control  over 
the  data  integrity  of  the  I£AR  and  ISD 
data  stores  (data  bases)  with^’ji  the 
IWSDB.  Only  the  "owners"  of  data 
bases  within  the  IWSDB  havo  the 
authority  to  write  to  t’  -  data  base. 
Such  authority  nay  be  delegated. 

(Note:  There  is  no  Process  004,  Data 
Store  D004 ,  or  Data  Store  D005 . ) 

Process  001  -  r^rgduce  Input 
Interchange  ^jies  -  The  LSAR  "owner" 
produces  th.j  input  interchange  files. 
This  activity  consists  of  preparing 
the  input  files  in  accordance  with  the 
approved  MII/-STD-1388-2B  data 
definitions  and  structure,  by 
e'i:racting  trcdning-related  data 
elements  from  the  LSAR.  In  addition 
to  producing  the  input  interchange 
files,  the  ISAR  "owner"  has  the 
responsibility  of  determining  changes 
from  the  previous  input  interchange 


files.  This  function  consists  of 
identifying  I£IAR  subsystem  and  task 
additions,  changes,  and  deletions. 

This  process  will  be  redesigned  upon 
fineil  ajprovcil  of  MIL-STD-1388-2B.  In 
the  interim,  the  process  has  been 
developed  for  MIL-STD-1388-2A-defined 
ISAR  data. 

Process  002  -  Transfer  Input 
Interchange  Files  -  The  DSS  Data  Base 
Administrator  (DBA)  transmits  the 
Irput  Interchange  Data  fron  the  IWSDB 
to  the  DSS  on  the  PC  subsystem  using  a 
standard  off-the-shelf  comunications 
package.  In  the  absence  of  a 
ccniiunications  procedure,  data 
transfer  can  take  place  over  an  "air 
gap"  using  tcpes  or  disks.  The  DBA 
transfers  the  data  to  hard  disk 
storage  in  data  store  D003  -  Input 
Interchange  File  Inage,  on  the  DSS  PC 
subsystem. 

Process  003  -  Translate  1388-2B  to  DSS 
Internal  Fomat  -  The  MIL-STD- 1368-28 
to  -2A  translation  of  data  elements 
only  needs  to  be  performed  if  MIL-STD- 
1388-2B  data  is  feeding  a  1388-2A- 
conforming  DSS  version.  Process  003 
will  not  be  fully  defined  until  the 
final  MIL-STD-1388-2B  is  ^proved. 

Hie  process  creates  data  store  D006  - 
DSS  Change  Files,  and  will  be 
performed  for  the  initial  load  of  ISAR 
data  and  eill  subsequent  updates. 

Process  005  -  Perform  IfiAR  Update  - 
Hiis  process  takes  place  only  if  new 
I£AR  data  has  been  downloaded  to 
ipdate  or  append  a  previous  download 
for  a  weapon  system.  Data  from  D006  - 
DSS  Change  Files,  will  be  identified 
as  additions,  changes,  and  deletions 
in  the  DSS.  Each  addition,  change, 
and  deletion  has  been  previously 
tagged  within  the  IWSDB  so  that  a  DSS 
user  will  have  the  option  of  inporting 
the  new  record  into  the  ancilysis.  The 
DSS  DBA  performs  the  initial  ISAR  data 
download.  'Hie  DSS  'Training 


Develcpment  Manager  and  Subsystem  Lead 
Analyst  interacts  with  ISAR  data 
updates  by  adding  subsystems/tasks 
Hirect.ly  into  the  DSS,  linking  an  ISAR 
addition  to  a  user-created 
subsystem/ task,  and  verifying  changes 
and  deletions. 

Process  006  -  Perform  ISP  Analysis  - 
All  DSS  ISD  analysts  perform  this 
process.  Each  analyst  interacts  with 
D007  -  DSS  Data  (ISAR  and  other 
analysis  data),  to  develop  an 
equipment  and  task  hierarchy. 

Analysts  then  determine  training 
system  requirements,  supported  by 
structured  decision  aids.  'Throughout 
the  ISD  analysis,  iterative 
read/writes  will  take  place  with  data 
store  D007. 

Process  007  -  Generate  Output 
Interchange  Files  -  The  DSS  DBA 
performs  the  process  of  creating 
Output  Interchange  Files  (Data  Store 
D008,  discussed  below)  through  a 
standard  DSS  report  c^ion.  'The 
output  format  facilitates  placement  of 
DSS  analysis  results  into  the  ISAR 
residing  within  the  IWSDB  (see  Process 
009). 

Process  008  -  'Transfer  Output 
Interchange  Files  -  'This  process 
resembles  process  002  -  Transfer  Input 
Interchange  Output,  but  transmits  data 
from  the  DSS  to  the  IWSDB  using  a 
standard  off-the-shelf  ccnnunications 
package.  In  the  absence  of  a 
ccmiunicatic^is  procedures,  data 
transfer  can  take  place  over  an  "air 
gap"  using  tapes  or  disks. 

Process  009  -  Transfer  Data  Fran 
Output  Interchange  Files  -  'Hiis 
process,  like  parocess  001  -  Produce 
Ir^t  Interchange  Files,  takes  place 
outside  of  the  DSS.  'The  actual 
interaction  of  the  ISAR  aixl  ISD  data 
"owners"  and  data  stores  D009  and  DOOl 
is  uniquely  defined  by  each  user. 


Data  contained  in  the  Output 
Interchcuige  Files  is  provided  to 
support  other  concurrent  logistics  and 
system  engineering  activities. 

The  following  sections  describe  Joint 
Service  ISD/LSAR  DSS  CMS  interface 
data  stores.  Hie  descriptions 
correspond  to  the  data  stores  on 
Figure  3. 

Data  Store  DOOl  -  ISAR  on  IWSDB  (1388- 
2B>  -  Hie  approved  MIL-STD-1388-2B 
will  prescribe  the  data  element 
definitions  (DEDs),  data  field 
lengths,  and  fomats  for  the  ISAR. 
Government  and  contractor 
organizatiCTis  will  be  required  to 
adhere  to  the  standard  for  all 
system/equipment  acquisition  programs, 
major  modification  programs,  and 
applicable  research  and  development 
projects  through  cill  phases  of  the 
system/equipment  life  cycle.  The 
physical  inplementation  of  the  MEL- 
STD-1388-2B  ISAR  data  will  vary  widely 
amcHig  users.  Hcwever,  Government  and 
contractor  organizations  will  be 
Cc^iable  of  producing  standard  ISAR 
output  reports.  The  Irput  Interchange 
Data  (data  store  D002),  required  to 
feed  ISAR  data  to  the  DSS,  have  been 
reccmnended  for  inclusion  in  the  final 
MIL-STD-1388-2B  as  a  standard  output. 

Data  Store  D002  -  Input  Interchange 
Data  (IWSDB)  -  Training  analyses  using 
the  DSS  do  not  require  ISAR  data,  but 
are  most  effective  v^ien  fully 
supported  by  avcdlable  ISAR  data. 

Only  training-related  ISAR  data 
elements  are  inported  for  ISD  analyses 
using  the  DSS.  Hie  ISAR  "owner" 
generates  this  data  store  for  ISAR 
data  transmittcil  to  the  DSS  PC 
subsystem.  Hie  Irput  Interchange  Data 
files  eue  tentatively  defined  pending 
final  approved  of  MIL-STD-1388-2B. 

Data  Store  0003  -  Input  Interchange 
Files  Image  -  This  data  store  mirrors 


D002  -  Iiput  Interchange  Data  (IWSIB), 
described  above.  When  D002  is 
transferred  frem  the  IWSDB,  D003 
exists  within  the  DSS  on  the  PC. 

Data  Store  D006  -  DSS  Change  Files  - 
Within  the  DSS,  D003  -  Iiput 
Interchange  Files  Image,  must  be 
translated  into  a  structure  that 
conforms  to  the  DSS  data  model 
(Process  003).  Hie  result  is  data 
store  D006.  Hiroughout  an  ISD 
anedysis,  the  Training  Development 
ffenager  and  Subsystem  Lead  Anedyst  may 
access  the  additicxis,  changes,  and 
deletions  as  required.  When 
exchanging  ISAR  data  with  the  [MEL- 
SH)-1388-2A  conforming]  DSS  Version 
4.0,  D006  files  cure  exactly  the  same 
cis  the  existing  ISAR  update  files. 

When  the  DSS  is  modified  to  conform 
with  an  approved  MEL-STD-1388-2B,  no 
translaticxi  is  anticipated  between  the 
D003  -  Irput  Interchange  Files,  and 
the  D006  -  DSS  Change  Files.  D003 
will  be  the  direct  irput  to  the  1388- 
2B  DSS,  and  Process  003  will  be 
eliminated. 

Data  Store  D007  -  DSS  Data  -  DSS 
Data  is  the  collection  of  data 
residing  in  the  Joint  Service  ISD/ISAR 
DSS.  Hie  data  include  ISAR  data, 
inserted  anedysis  administrative  data, 
training  system  requirements 
decisions,  and  analysis  working  data. 
Hie  DSS  DBA,  Tredning  Development 
(femager,  and  ISD  Anedysts  access  this 
data  store  continually  throughout  an 
anedysis. 

Data  Store  D008  -  Output  Interchange 
Files  -  Output  Interchange  Files, 
consists  of  MIL-STD-1388-2B  and  MEL- 
STD-1379D  data  elements  resulting  from 
ISD  analyses  performed  using  the  DSS. 
These  data  eure  for  use  by  the  ISAR 
"owner"  and  other  concurrent  logistics 
and  engineering  processes,  thus 
sufporting  the  intent  of  the  DoD  CALS 
initiatives. 
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Data  Store  D009  -  CXitput  Interchange 
Data  (IWSDB)  -  This  data  store  mirrors 
Data  Store  D008  -  Output  Interchange 
Files  within  the  IWSDB. 

P-ATfi  Interface  Develoanent  Findings 
and  Reccmnendations  -  To  design  an 
ISD-ISAR  data  interface  that  is  both 
CALS  conopatible  and  operationally 
practical,  user  involvement  in  the 
design  process  Wcis  essential.  The 
Joint  Service  ISD/ISAR  DSS  CALS 
interface  design  and  operational 
concept  were  discussed  at  two  Joint 
Service  ISD/I£!AR  DSS  CAI£  interface 
working  group  (IW3)  meetings.  The  IWG 
meetings  served  cis  a  forum  in  v^ch 
expec±ed  users  participated  in  the 
interface  concept  developnent  and 
design.  The  IWSs  tcxDk  place  on  23-24 
May  1990  and  10-12  July  1990,  in 
Wcishington,  DC.  A  number  of  DSS  CAIS 
interface  design  and  inplementation 
issues  were  identified  and  discussed; 

Issue  # 

001  MIL-STD-1388-2B  Status 

002  Ongoing  Evolution  of  CALS 
Training  Data  Element 
Definitions  and  Relationships 
003  Input  Interchange  File 

Construction  by  IBAR  "Comers" 
004  Output  Interchcinge  File  Review 
and  Handling  by  IBAR  cind  ISD 
Data  "(Xmers" 

005  Concurrent  Government  and 

Contractor/Sub-contractor  Use 
of  Interchange  Files 
006  LSAR  Update  Data  Coiparisons 

007  CAIB  Interchange  Data 

Concurrency 

008  Data  Translation  fron  MIL-STD- 
1388-2B  to  DSS  Format 
009  September  1990  CALS  Interface 
Demonstration 

010  Ongoing  ISD/IBAR  DSS  CAIB 
Interface  Information 
Exchange 

Detailed  discussions  of  each  interface 


design/implemBntaticxi  issue  are 
documented  in  the  Joint  Service 
ISD/IBAR  DSS  CAIB  Interface 
Rpqnir«=ments  Report.  (Final).  Dynamics 
Research  Corporation,  31  July  1990. 

Extensive  user  iiput  on  each  CALS 
interface  design/ inplementation  issue 
resulted  in  an  interface  design  that 
is  responsive  to  needs  of  both 
Government  cuid  industry  Joint  Service 
ISD/IBAR  DSS  users.  Moving  beyond  the 
limitations  of  a  CALS  (Phase  1) 
solution  is  a  desired  abjective  that 
will  require  additional  work.  Final 
approval  of  MrL-STD-1388-2B  is  a 
necessary  prerequisite  to  a  Joint 
Service  ISD/IBAR  DSS  CAIB  (Phase  2) 
interface.  The  following  three 
findings  sunrarize  the  Joint  Service 
ISD/IBAR  DSS  CALS  interface 
develcpxnent  work  performed  to  date: 

Finding  #1;  An  "evolving"  MTL-STD- 
1388-2B  (and  other  data  element 
standards)  will  mandate  the  periodic 
revision  of  the  architecture  and/or 
the  data  interfaces  of  analysis  tools 
that  use  data  defined  in  the  standard. 

Finding  #2;  A  CAIB  (Phase  2)  IBAR 
data  interface  would  overccme  nany  of 
the  inefficiencies  and  data  integrity 
problems  that  exist  in  the  Joint 
Service  ISD/IBAR  DSS  CAIB  (Phase  1) 
interface.  CAIB  (Phase  2)  interfaces 
may  ultimately  prove  to  be  unique 
arrangements  of  tools  and  data  bases, 
rather  than  generalized  solutions. 

Finding  #3;  Hie  ability  to  track  IBAR 
data  changes  and  assess  their  impacts 
on  training  system  design  is  an 
attractive  feature  of  the  Joint 
Service  ISD/IBAR  DSS.  Hie  limited 
current  cepability  could  be  expanded 
significantly  with  the  use  of  MIL-STD- 
1388-2B  data  element  date/time 
staiping  or  eiltemative  schemes  to 
track  IBAR  changes. 


In  response  to  the  three  findings^  two 
general  reccnmendations  were  itede: 

(1)  Make  the  effort  to  get  data 
element  standards  "right  the 
first  time.”  Balance  any  need 
to  siabsequently  modify  the 
standard  with  the  cost  and 
effort  required  to  modify 
infomation  systems  that  use  the 
data. 

(2)  Use  date/time  stairping  or 
alternate  schemes  to  track  data 
element  changes  in  order  to 
facilitate  the  efficient 
assessment  of  design  changes  on 
all  related  logistics  processes 
and  system  engineering 
activities. 


airnnary 

The  Joint  Service  ISD/LSAR  DSS  is  a 
powerful  tool  for  performing  ISD 
aricilyses  of  weapon  systems.  The 
autoiated  capabilities  eliminate 
labor-intensive  data  handling  tasks 
and  allcw  training  analysts  to 
effectively  focus  on  the  cuicilysis  of 
training  system  requirements.  With 
its  CAI£  cdipatible  ISAR  data 
interface,  the  Joint  Service  ISD/ISAR 
DSS  offers  nary  advantages  over  other 
ISD  analysis  methods.  The  CAIS 
interface  generates  productivity 
inproveraents  by  including  training 
system  development  in  integrated 
product  development/concurrent 
engineering.  The  effective  use  of 
aiva liable  engineering/logistics  data 
generates  time  and  cost  savings, 
inproves  decision  naking,  and  si:pports 
life  cycle  product  inprovements. 
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Instructional  Systems  Development  Automation 

MSgt  Samuel  D.  Howard 
MSgt  Eric  L.  Diel 
TSgt  Clarence  w.  Poteat 
TSgt  William  E.  Reid 

The  cold  war  has  ended  and  a  stable,  democratic  peace  is  spreading 
across  the  European  continent.  The  Berlin  Wall  has  come  down,  but  another 
wall  still  remains — the  wall  of  labor  intensive  instructional  systems  design. 
This  wall  blocks  our  objectivity  and  clouds  our  vision  of  true  system  training 
requirements .  But  we  will  vanquish  the  wall  with  our  own  olive  branch  process 
-  Instructional  Systems  Development  Automation. 

Instructional  Systems  Development  Automation  (ISDA)  is  the  name  of  the 
relational  database  program  we  have  developed  to  address  the  dual  problems  of 
labor  intensiveness  and  lack  of  standardized  analysis  parameter  application. 

ISDA  addresses  the  first  problem  by  eliminating  repetitive  data  input, 
filtering  out  non-training  requirements  early  in  the  analysis  process,  and 
automating  report  generation.  The  analyst  inputs  data  from  any  source,  i.e., 
LSA,  technical  manuals,  etc.  This  information  is  broken  down  into  sequential 
activities  and  analyzed.  The  software  uses  embedded  decision  logic  to  pull 
forward  only  those  activities  that  require  further  analysis.  The  system 
generates  a  variety  of  reports,  from  task  and  duty  lists  to  hardware  and  CBT 
fidelity  analysis  documentation.  These  reports  had  consumed  a  considerable 
amount  of  consolidation  effort  under  the  "paper"  analysis  methodology. 

ISDA  addresses  the  second  problem  by  using  embedded  decision  logic  to 
assist  the  analyst  in  making  training  decisions.  The  system  prompts  the 
analyst  with  questions  at  the  task  activity,  knowledge  and  skilled  behavior, 
media,  hardware,  and  CBT  fidelity  analysis  levels.  The  analyst's  answers  to 
these  questions  provide  the  relevant  information  that  the  system  uses  to 
baseline  decisions.  The  bottom  line  is  that  the  system  focuses  the  analysis 
and  allows  the  Subject  Matter  Expert  to  develop  straightforward,  objective, 
training  recommendations . 

MSgt  Howard  is  the  chief.  Training  systems  and  Technology  Branch,  3306th 
Training  Development  Squadron.  He  holds  an  Associate  of  Applied  Science 
degree  in  Avionics  Technology.  He  is  currently  involved  in  researching  and 
developing  applications  of  technology  for  developing  and  conducting  psychomo¬ 
tor  and  cognitive  skills  training  and  education. 

MSgt  Diel  was  a  Training  Systems  Analyst  with  the  3306th  Training  Development 
Squadron  at  the  time  this  paper  was  written.  He  is  now  the  Assistant  Director 
of  the  Edwards  Air  Force  Base  Family  support  center.  He  holds  a  Bachelor  of 
Arts  Degree  in  Liberal  Arts. 

TSgt  Poteat  is  a  Training  systems  Analyst  for  the  3306th  Training  Development 
Squadron.  He  holds  two  Associate  degrees:  Avionics  Technology  and  Instructor 
of  Technology.  He  is  our  resident  expert  in  applying  ISD. 

TSgt  Reid  is  a  Systems  Analyst  for  the  3306th  Training  Development  Squadron. 
Completely  self-taught  in  computers,  he  wrote  our  ISD  Automation  software. 
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Instructional  systems  Development  Automation  (ISOA) 

MSgt  Samuel  D.  Howard 
MSgt  Eric  L.  Diel 
TSgt  Clarence  w.  Poteat 
TSgt  William  E.  Reid 

These  two  problems  started  the 
330bth  on  its  development  of  ISDA. 


IHTR^DUCTlCTi 

A  significant  part  of  the  3306th 
Training  Develoj^ment  Squadron's 
mission  is  to  determine  maintenance 
training  requirements  for  new  or 
highly  modified  weapon  systems.  To 
this  end  we  use  a  "tailored"  ap¬ 
proach  to  Instructional  Systems 
Development  (ISD) .  The  model  for 
this  process  was  developed  in  1979 
by  Applied  Science  Associates  under 
contract  to  the  Air  Force  Human 
Resources  Laboratory  (Now  Armstrong 
Laboratory) .  Applied  Science  Asso¬ 
ciates  delivered  a  closed  14  step 
model  for  front  end  analysis  and 
ISD.  The  model  has  served  the 
squadron  and  Air  Force  very  well  but 
it  also  has  two  inherent  problems. 
The  first  problem  is  that  the  14 
step  paper  model  is  very  labor 
intensive.  It  requires  the  analyst 
to  document  training  decisions  on  a 
series  of  forms.  The  training 
decisions  are  transferred  from 
document  to  document,  creating  a 
tremendous  "paper  trail"  for  each 
maintenance  task  under  analysis. 
After  several  months  of  an.  lysis 
this  trail  evolves  into  a  pile  of 
such  proportions  that  it  is  almost 
impossible  for  the  analyst  to  accu¬ 
rately  update  the  analysis  as  new 
information  becomes  available.  The 
second  problem  is  one  of  applica¬ 
tion.  The  14  step  model  involves 
decision  algorithms.  The  analyst 
applies  algorithms  to  the  task  under 
analysis  and  makes  training  recom¬ 
mendations  based  on  the  decis  on 
logic.  The  problem  is  that  analysts 
don't  apply  the  algorithms  consist¬ 
ently.  What  is  new  to  one  may  not 
be  new  to  another.  What  seems  to  be 
hardware  cues  to  one  analyst  may  not 
be  to  another.  The  list  goes  on. 


PROCESS 

The  process  has  evolved  from  the 
original  14  step  model,  through  a  13 
step  process,  to  the  15  ste^  model 
that  we  have  today.  The  following  is 
a  brief  overview  of  each  of  the  15 
steps . 

Step  1  -  Identify  System  Maintenance 
Requxi'aients 

In  this  step  the  analyst  identifies 
all  duties  and  tasks  that  must  be 
performed  to  keep  the  weapon  system 
operational.  The  analyst  also  gath¬ 
ers  all  relevant  weapon  system  data. 

Step  2  -  Identify  Characteristics  of 
the  Target  Population 

The  analyst  identifies  the  charac¬ 
teristics  of  the  target  population 
that  could  have  a  bearing  on  train¬ 
ing  requirements .  Target  population 
previous  training,  skills,  knowl¬ 
edge,  and  experience  are  documented. 
This  constitutes  the  analysis  base¬ 
line  and  course  ’■  ntry  prerequisites. 

Step  3  -  Determine  Task  Based  Train¬ 
ing  Requirements 

This  step  requires  that  the  analyst 
break  each  task  identified  in  Step  1 
down  into  its  specific  activities. 
The  analyst  then  measures  the  activ¬ 
ities  against  specific  criteria.  If 
the  activity  meets  any  one  cr  more 
of  the  analysis  criteria,  it  is 
identified  as  a  potential  training 
requirement.  All  potential  training 
requirements  are  carried  forward  for 
further  analysis. 
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All  non-training  steps  are  flagged 
as  "integrate",  which  means  that 
they  may  be  included  in  the  training 
program  but  they  won't  drive  any 
speci'lc  hardware  or  material  re- 
quire-.ents . 

Those  activities  that  drive  poten¬ 
tial  training  requirements  are 
pulled  into  Behavior  Analysis.  In 
this  sub-step  the  analyst  breaks  the 
activities  down  into  component 
knowledges  and  behaviors .  The 
knowledges  and  behaviors  are  then 
evaluated  against  specific  analysis 
parameters.  If  the  knowledge  or 
behavior  meets  any  one  or  more  of 
the  behavior  analysis  criteria,  it 
becomes  a  firm  training  requirement. 
As  soon  as  a  specific  knowledge  or 
behavior  is  flagged  as  a  training 
requirement,  the  analyst  assigns  a 
taxonomic  code  to  categorize  the 
type  of  learning  that  is  going  on. 

Step  4  -  Determine  Concept  Based 
Training  Requirements 

Step  3  required  the  analyst  to 
identify  all  task  specific  training 
requirements.  Many  tasks  require 
that  the  technician  have  prerequi¬ 
site  knowledge  to  be  able  to  perform 
accurately.  Fault  isolation  tasks, 
inspections,  and  operational  checks 
are  major  categories  that  require 
the  technician  have  a  firm  under¬ 
standing  of  the  concepts  associated 
with  the  task  performance  require¬ 
ment.  This  step  is  dedicated  to 
determining  those  types  of  require¬ 
ments.  Tasks  containing  knowledge 
or  skilled  behavior  training  re¬ 
quirements  that  meet  a  specific 
taxonomic  threshold  are  pulled 
forward  for  concept  analysis.  The 
analyst  determines  which  concepts 
are  associated  with  the  task  knowl¬ 
edge  or  skilled  behavior  training 
requirements.  Once  identified,  the 
concepts  are  broken  down  into  their 
elements.  The  elements  are  analyzed 
against  specific  criteria.  If  the 
element  meets  any  one  or  more  of  the 
factors  it  is  a  potential  training 


requirement.  The  potential  concept 
element  training  requirements  are 
subsequently  broken  down  into  their 
discrete  characteristics.  The 
characteristics  are  then  analyzed 
against  specific  criteria.  If  the 
characteristic  meets  any  of  the 
analysis  parameters,  it  must  be 
trained. 

Step  5  -  Determine  Media  and  Method¬ 
ology 

This  step  requires  that  the  analyst 
identify  a  specific  media  class  for 
each  task  knowledge,  skilled  behav¬ 
ior,  and  concept  based  training 
requirement  identified  in  the  previ¬ 
ous  steps.  The  analyst  first  deter¬ 
mines  the  method  of  transmission  for 
the  instructional  message.  The 
analyst  then  determines  the  domain 
of  learning  that  is  being  addressed. 
This  leads  the  analyst  into  specific 
media  analysis  algorithms  which 
assist  the  analyst  in  determining 
the  appropriate  media  class  for  the 
behavioral  or  concept  based  training 
requirement.  The  media  classes  range 
from  hardware  to  self-paced  study 
guides.  The  analyst  selects  the 
method  of  instruction  best  suited  to 
the  specific  media  requirement. 

Step  6  -  Develop  Instructional 
Strategies 

In  this  step  the  analyst  develops  a 
criterion  objective  for  each  task  or 
concept  based  training  requirement . 
In  addition,  the  analyst  develops  a 
criterion  referenced  test  for  each 
criterion  objective.  The  analyst 
describes  and  links  the  training 
requirements  to  specific  media 
descriptions.  The  analyst  develops 
an  instructional  strategy  that  tells 
how  the  media  will  be  employed  to 
accomplish  the  criterion  objective. 
The  analyst  sequences  all  task-  and 
concept-based  education  and  training 
requirements . 
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step  7  -  Identify  Fidelity  Require- 
nents  of  Hardware  CoHqxinents 

In  step  5  the  analyst  determines  the 
classes  of  media  that  are  required 
to  support  the  training  require¬ 
ments  .  Step  7  requires  that  the 
analyst  get  very  specific  and  iden¬ 
tify  the  components  and  required 
fidelity  levels  for  the  hardware 
devices.  To  do  this  the  SHE  will 
look  at  each  of  the  behavioral 
training  requirements  that  drive 
hardware.  The  analyst  identifies 
the  specific  hardware  components 
associated  with  the  behavior.  The 
analyst  will  then  analyze  the  compo¬ 
nent  against  functional  and  physi¬ 
cal  fidelity  algorithms.  The  pur¬ 
pose  is  to  determine  the  level  of 
fidelity  (or  realism)  required  to 
facilitate  proper  training  and 
transfer.  The  analyst  also  deter¬ 
mines  "whole  panel"  fidelity  re¬ 
quirements,  which  allows  for  maximum 
psychological  fidelity  in  the  train¬ 
ing  device. 

Step  8  -  Identify  Fidelity  Require- 
Bents  of  Ccm^ut'r  Based  Training 
(CBT) 

During  the  media  analysis  that  is 
performed  in  Step  5,  the  analyst 
might  find  that  CBT  is  the  appropri¬ 
ate  media  class  for  some  of  the 
training  requirements.  In  step  8 
the  analyst  determines  the  required 
levels  of  fidelity  for  all  training 
requirements  that  drive  CBT.  CBT 
can  span  the  range  of  simple  text 
based  tutorial  to  high  end  interac¬ 
tive  video  with  unlimited  free  play. 
The  purpose  of  the  step  is  to  deter¬ 
mine  the  most  training  effective  CBT 
class  to  meet  the  requirement.  The 
analyst  will  measure  each  behavior 
and  concept  characteristic  that 
drove  the  CBT  class  against  a  CBT 
fidelity  algorithm.  The  level  of 
fidelity  is  determined  as  a  result 
of  the  analysis. 


Step  9  -  Select  Instructional  Fea¬ 
tures 

There  are  four  aspects  of  any  educa¬ 
tion  or  training  scenario.  They 
are:  stimulus,  response,  feedback, 
and  next  activity.  In  the  tradi¬ 
tional  instructor-led  scenario,  it's 
the  instructor  who  supervises  all 
four  aspects.  However,  it's  possi¬ 
ble  for  hardware  to  be  designed  to 
control  some  or  all  of  the  training 
or  education  environment.  This  is 
why  we  determine  instructional 
features  for  hardware  media.  The 
3306th  ISD  process  guides  the  ana¬ 
lyst  through  two  series  of  decision 
algorithms.  The  first  series  of 
algorithms  is  used  to  determine  what 
should  control  the  learning  environ¬ 
ment  for  specific  hardware  driving 
tasks.  The  analyst  uses  the  second 
series  of  algorithms  to  determine 
the  specific  instructional  feature 
requirements  for  all  training  re¬ 
quirements  that  the  hardware  train¬ 
ing  system  should  control. 

step  10  -  Prepai-e  ISD-Derived  Train¬ 
ing  Equipment  Specifications 

-  The  training  equipment  functional 
specification  is  the  vehicle  that 
conveys  the  weapon  hardware  training 
equipment  and  CBT  (hardware  only) 
requirements  to  the  Program  office. 
The  specification  is  a  compilation 
of  all  analysis  data  and  is  organ¬ 
ized  into  five  major  paragraphs. 

They  are:  Training  objectives.  Train¬ 
ing  Application,  simulation  Charac¬ 
teristics,  Instructional  Features, 
and  Trainer  Configuration. 

steps  11  Through  15 

The  remaining  steps  in  the  3306th 
TDS  ISD  methodology  are  Step  11, 
Prepare  Course  Control  Dor-jnents; 

Step  12,  Prepare  Instructional 
Materials;  Step  13,  Validate  In¬ 
struction;  Step  14,  Conduct  Train¬ 
ing;  and  Step  15,  Evaluate  Training. 
The  3306th  performs  these  steps  as 
in  the  traditional  five  step  ISD 
model . 
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PROGRAM  STRacrrURB 

The  Instructional  systems  Develop¬ 
ment  Automation  (ISDA)  software  is 
designed  to  automate  the  3306th  TDS 
ISD  process  through  step  11,  Prepare 
Course  Control  Documents .  The  soft¬ 
ware  is  coded  in  Clipper,  a  rela¬ 
tional  database  programming  lan¬ 
guage.  It  will  run  on  any  IBM  PC  or 
compatible  with  512K  RAM  and  DOS  2.0 
or  higher.  It  will  run  from  a 
floppy  diskette  but  a  hard  drive  is 
recommended  because  the  data  files 
grow  very  quickly.  The  program  is 
built  in  16  modules  that  basically 
mirror  the  3306th  ISD  process.  These 
are;  Analysts,  Duties,  Target  Popu¬ 
lation,  Tasks,  Activity  Analysis, 
Knowledge/Skilled  Behavior  Analysis, 
Concepts,  Element  Analysis,  Charac¬ 
teristic  Analysis,  Media  Analysis, 
Instructional  Strategies,  Media 
Description,  Sequencing,  Hardware 
Fidelity,  CBT  Fidelity,  and  Instruc¬ 
tional  Features . 

Analysts  Module 

The  training  manager  or  each  analyst 
manually  inputs  the  data  pertaining 
to  the  analyst,  and  the  respective 
weapon  subsystem(s)  that  will  re¬ 
quire  examination  on  the  system. 

The  analyst  database  provides  iden¬ 
tification  data  for  the  various 
reports  in  ISDA. 

Duties  Module 

The  analyst  manually  inputs  the  data 
pertaining  to  the  specific  duties 
the  technician  will  be  required  to 
perform  on  the  system.  The  analyst 
must  build  a  duties  database  to 
start  the  analysis.  The  duty  list 
focuses  the  analysis  for  the  devel¬ 
opment  task  list  that  follows.  The 
analysis  proceeds  as  illustrated  in 
Figure  1 . 

Target  Population  Nodule 

The  TARGET  POPULATION  module  is 
similar  to  the  previous  modules  in 


that  the  data  is  entered  manually 
and  no  decisions  are  made  by  the 
system.  The  specific  purpose  of 
this  module  is  to  allow  the  analysts 
to  enter  specific  information  about 
their  target  audience.  The  goal  is 
to  focus  the  analysts'  attention. 
Though  the  software  does  not  use  the 
information  entered  into  this  module 
for  any  specific  decisions  or  ac¬ 
tions,  its  significance  cannot  be 
taken  for  granted.  Every  training 
and  education  decision  that  the 
analyst  will  make  in  the  following 
steps  is  based  on  the  target  audi¬ 
ence  . 

Task  Module 

The  TASK  database  follows  the  duty 
database.  The  task  titles  are  en¬ 
tered  manually,  when  building  the 
task  database  the  system  requires 
the  analyst  to  group  tasks  under  the 
previously  defined  duties.  The 
duties  in  essence  become  major 
categories  of  work  or  activity.  The 
3306th  has  found  that  this  "struc¬ 
turing”  of  the  analysis  allows  for  a 
more  logical  and  organized  analysis. 

Activity  Analysis  Module 

The  fourth  module  in  the  ISDA  soft¬ 
ware  is  ACTIVITY  ANALYSIS.  This 
module  draws  in  the  previously 
identified  tasks  and  prepares  them 
for  analysis.  The  analyst  must 
manually  enter  each  of  the  activi¬ 
ties  (also  referred  to  as  steps) 
that  make  up  each  task.  Upon  enter¬ 
ing  the  activity,  the  system  will 
open  the  activity  analysis  field. 
This  is  where  the  formal  decision 
process  leading  to  training  and 
education  requirement  determination 
begins.  This  field  contains  the  six 
parameters  against  which  the  analyst 
will  measure  the  task  activities. 

The  parameters  are:  new,  complex, 
condition,  criteria,  negative  trans¬ 
fer,  and  tool /equipment .  The  soft¬ 
ware  is  designed  to  query  the  ana¬ 
lyst  with  a  series  of  "yes /no"  type 
questions  for  each  analysis  parame- 
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ter.  The  questions  are  designed  to 
determine  if  in  fact  a  specific 
activity  is  new  to  the  target  popu¬ 
lation,  if  it  is  cognitively  complex 
for  the  audience,  if  specific  condi¬ 
tions  that  inhibit  performance  of 
psychomotor  skills  are  present,  etc. 
This  allows  the  analyst  to  focus  on 
performing  as  a  subject  matter 
expert  and  increases  the  objectivity 
and  consistency  of  the  recommenda¬ 
tions.  The  ACTIVITY  ANALYSIS  module 
is  also  where  the  filtering  of 
potential  training  requirements 
versus  non-training  requirements 
begins.  If  the  analyst  determines 
that  none  of  the  six  analysis  param¬ 
eters  applies  to  the  activity,  the 
activity  is  flagged  "Integrate 
only" .  This  means  that  the  item 
requires  no  further  analysis.  It 
won't  be  pulled  forward  into  behav¬ 
ior  analysis.  Activities  flagged  as 
integrate  can  be  included  into  a 
specific  task  training  requirement 
if  the  analyst  feels  that  they  are 
required  for  task  continuity  or 
positive  transfer.  Integrated 
activities  will  not  have  any  media 
resources  dedicated  specifically  to 
them. 

Knowledge/SldLlled  Behavior  Analysis 
Module 

The  fifth  module  contains  the  code 
used  to  determine  knowledge  and 
skilled  behavior  training  require¬ 
ments.  Each  activity  that  is  a 
potential  training  requirement  is 
automatically  pulled  forward.  The 
analyst  determines,  and  manually 
inputs,  the  discrete  knowledge  and 
skilled  behaviors  that  make  up  the 
activity.  Each  is  analyzed  against 
the  Seime  six  analysis  parameters 
used  in  activity  analysis;  the 
system  queries  the  analyst  to  en¬ 
hance  objectivity.  If  a  knowledge 
or  skilled  behavior  falls  out  as 
true  against  one  or  more  of  the 
parameters,  the  system  stores  the 
unit  as  a  training  requirement. 
Finally,  the  analyst  categorizes  the 
training  requirement  by  using  a 


learning  taxonomy  classification. 

The  system  prompts  the  analyst  for 
the  taxonomy  by  providing  a  list  of 
relevant  action  verbs  for  each 
taxonomy.  The  learning  taxonomies 
used  by  the  system,  in  hierarchal 
sequence  are:  Associating,  Recalling 
Facts  &  Principles,  Recalling  Proce¬ 
dures,  Repetitive  Movement,  continu¬ 
ous  Movement,  Positioning  &  Serial 
Movement,  Discriminating,  Classify¬ 
ing,  Rule  Using,  and  Problem  Solv¬ 
ing. 

Concept  Analysis  Nodule 

Concept  Analysis  is  directly  linked 
to  the  previous  task  analysis  mod¬ 
ule.  This  section  is  based  on  the 
theory  that  categories  or  levels  of 
psychomotor  activity  require  levels 
of  understanding  of  specific  con¬ 
cepts  relating  to  the  system. 

ISDA  determines  if  there  is  a  poten¬ 
tial  requirement  for  concept  analy¬ 
sis  by  scanning  the  previously 
completed  task  database  and  all  of 
the  corresponding  task  activity 
knowledge  and  skilled  behavior 
training  requirements.  If  any  of 
the  requirements  meet  the  taxonomic 
threshold  of  classification  the 
entire  task  is  automatically  pulled 
forward  into  the  concept  analysis 
database.  The  system  then  queries 
the  analyst  to  determine  if  concept 
analysis  must  be  performed  for  the 
task.  If  the  analyst  determines  that 
concept  analysis  is  required,  the 
system  will  prompt  for  a  concept 
title. 

Elenent  Analysis  Nodule 

Once  entered,  the  analyst  will  break 
the  concept  down  into  specific 
elements .  The  elements  are  then 
analyzed  against  three  analysis 
parcimeters;  Relevant,  Incidental, 
and  Irrelevant.  At  this  point,  the 
analyst  is  determining  potential 
training  requirements  in  much  the 
same  way  as  was  done  previously  in 
task  activity  level  analysis.  The 
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system  prompts  the  analyst  with  a 
series  of  questions  to  determine 
where  the  element  falls  in  regard  to 
the  analysis  parameters.  If  an 
element  is  True  against  Relevant  or 
Incidental,  it  drives  a  potential 
training  requirement  and  will  be 
pulled  forward  for  further  analysis. 

Characteristics  Analysis  Module 

The  next  step  is  for  the  analyst  to 
break  each  potential  element  train¬ 
ing  requirement  down  into  its  dis¬ 
creet  characteristics.  The  charac¬ 
teristics  are  units  of  information 
that  are  specific  to  the  element, 
things  that  illustrate  the  element, 
make  it  easier  to  understand,  or 
easier  to  apply  on  the  job.  The 
characteristics  are  analyzed  against 
three  analysis  pareuneters:  New, 
Complex,  and  Critical.  The  system 
will  again  prompt  with  a  series  of 
questions  to  aid  the  analyst  in 
making  this  decision.  If  the  char¬ 
acteristic  is  true  against  any  of 
the  three  parameters  it  must  be 
included  in  the  training  program. 

For  those  elements  that  require 
training,  ISDA  will  go  into  media 
and  method  analysis.  The  media 
analysis  decision  logic  for  this 
portion  of  training  requirements 
will  be  restricted  to  the  tracks 
that  pertain  to  the  cognitive  do¬ 
main.  This  again  increases  the 
objectivity  of  the  training  deci¬ 
sions.  The  analyst  can  generate 
comprehensive  reports  of  task  and 
concept  analyses  that  show  training 
and  non-training  requirements, 
learning  taxonomy  classifications, 
and  media  requirements . 

Media  Analysis  Nodule 

once  a  task  knowledge,  skilled 
behavior,  or  concept  is  established 
as  a  training  requirement,  ISDA 
requires  that  a  specific  media  class 
and  method  of  instruction  be  as¬ 
signed  to  it.  The  media  analysis 
code  uses  embedded  decision  logic 
that  splits  media  analysis  into  six 


individual  tracks .  The  appropriate 
track  is  based  on  whether  the  re¬ 
quirement  is  instructor  or  medium 
based  and  which  domain  of  learning 
(cognitive,  affective,  psychomotor) 
the  requirement  resides  in.  The 
system  again  queries  the  analyst, 
and  determines  the  appropriate  media 
track,  and  ultimately  the  media 
class,  based  on  the  analyst's  an¬ 
swers.  The  method  of  instruction  is 
automatically  entered  by  ISDA  based 
on  the  media  class. 

Instructional  Strategies  Module 

The  system  scans  the  databases  for 
all  tasks  and  concepts  that  contain 
either  knowledges,  skilled  behav¬ 
iors,  or  characteristics  that  con¬ 
tain  training  requirements .  The 
software  carries  all  related  task  or 
concept  data  (i.e.  specific  training 
requirements,  media  requirements, 
etc.)  forward.  The  analyst  selects 
a  task  from  this  list  of  trainable 
tasks.  Then  the  system  prompts  the 
analyst  for  a  learning  behavior,  and 
an  instructional  strategy  for  spe¬ 
cific  use  of  media.  Finally,  the 
system  prompts  the  analyst  for  a 
teaching  time.  This  is  an  estimate 
on  the  part  of  the  analyst  on  how 
long  it  will  take  to  teach  and  test 
the  task  or  concept  based  training 
requirement.  These  figures  are  used 
to  calculate  lesson,  block,  and 
course  training  times . 

Media  Description  Nodule 

This  portion  of  the  analysis  pro¬ 
vides  a  detailed  description  of 
specific  media  requirements  that 
allows  the  media  developer  to  tailor 
specific  media  to  specific  require¬ 
ments.  The  system  will  prompt  the 
analyst  for  descriptions  of  each  of 
the  media  classes  identified  previ¬ 
ously  in  the  Behavior  or  Character¬ 
istics  Analysis  modules.  The  system 
will  have  the  analyst  enter  a  de¬ 
scription  of  specific  media,  and 
link  the  description  to  the  knowl¬ 
edge,  skilled  behavior,  or  charac- 
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teriat 3  c  training  requirements  with 
which  it  is  associated. 

Sequencing  Module 

This  involves  sequencing  the  train¬ 
ing  requirements  into  a  course 
structure.  The  system  will  prompt 
the  analyst  to  sequence  the  tasks 
and  concepts  to  be  trained  into 
lessons.  Then  the  system  prompts 
the  analyst  to  sequence  the  lessons 
into  blocks.  Finally,  the  blocks 
are  sequenced  into  courses . 

Hardware  Fidelity  Analysis  Module 

once  instructional  strategies  have 
been  completed,  the  analyst  can  move 
into  hardware  fidelity  analysis. 

The  system  scans  all  databases  and 
draws  forward  those  task  activities, 
knowledges,  and  skilled  behavioral 
training  requirements  that  are 
flagged  as  requiring  hardware.  The 
system  then  asks  the  analyst  to 
enter  the  names  of  the  hardware 
components  that  the  student  must 
interface  with  during  task  training 
and  performance.  Once  entered,  the 
system  will  query  the  analyst  for 
the  required  levels  of  functional 
and  physical  fidelity.  The  system 
bases  the  required  fidelity  levels 
on  the  analyst's  responses.  Once 
all  primary  hardware  items  have  been 
identified  and  analyzed,  the  system 
asks  the  analyst  to  identify  all 
"whole  panel”  components.  These  are 
components  which  students  interface 
with  indirectly.  Surround  compo¬ 
nents  on  a  control  panel,  i.e, 
switches,  lights,  and  bezels  are 
examples  of  whole  panel  components . 
While  not  significant  from  a  per¬ 
formance  perspective,  they  are 
important  to  the  psychological 
fidelity  of  the  training  system.  By 
maximizing  psychological  fidelity  as 
much  as  possible  without  overbuild¬ 
ing  the  trainer  we  enhance  transfer 
out  to  the  actual  system,  once  all 
components  have  been  entered  and 
analyzed  at  the  task  level,  the 
system  will  build  a  database  of  all 


components  and  recommended  fidelity 
levels.  The  system  will  show  the 
results  of  this  compilation  to  the 
analyst  and  ask  if  the  level  speci¬ 
fied  is  satisfactory.  If  so,  the 
analyst  accepts  the  fidelity  recom¬ 
mendation  and  moves  on.  If  not,  he 
can  modify  the  final  fidelity  recom¬ 
mendation.  The  system  will  generate 
a  complete  list  of  all  required 
hardware  items  that  must  be  included 
on  the  training  system  and  their 
required  fidelity  levels  in  table 
format  that  is  ready  for  inclusion 
into  the  training  equipment  func¬ 
tional  specification. 

Computer  Based  Training  Fidelity 
Analysis  Module 

The  system  is  also  designed  to  allow 
the  analyst  to  perform  Computer 
Based  Training  (CBT)  fidelity  analy¬ 
sis.  This  section  is  based  on  the 
fact  that  there  are  a  variety  of 
possible  levels  of  CBT.  They  span 
the  range  from  simple  text  based 
tutorials  to  fully  interactive  video 
simulations  with  numerous  branching 
options.  The  level  of  the  task  or 
concept  based  training  requirement 
determines  the  level  of  CBT  that  is 
most  appropriate.  The  system  will 
scan  all  previous  analysis  databases 
and  build  a  new  database  containing 
all  task  or  concept  based  training 
requirements  that  drive  CBT.  The 
system  will  show  each  task  or  con¬ 
cept  and  associated  knowledges, 
skilled  behaviors,  elements,  and 
characteristics  that  drove  the  CBT 
requirement.  The  system  will  then 
query  the  analyst  using  questions 
that  deal  with  the  type  of  learning 
and  technical  content  of  the  materi¬ 
al.  The  system  takes  this  input  and 
gives  a  recommendation  on  the  level 
of  CBT  that  would  be  most  appropri¬ 
ate.  The  analyst  can  accept  the 
recommendation  or  change  it  if 
circumstances  warrant. 

This  portion  of  the  progreun  will 
generate  a  CBT  fidelity  report  thut 
can  be  used  as  a  development  guide 
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for  the  CBT  courseware.  By  stand¬ 
ardizing  the  logic  that  goes  into 
making  the  decisions  we  achieve  a 
more  uniform  approach  to  the  course¬ 
ware  development  effort. 

Instructional  Features  Hodule 

The  final  analysis  module  in  ISDA 
deals  with  the  selection  of  instruc¬ 
tional  features  for  the  hardware 
training  system.  Traditional  train¬ 
ing  scenarios  have  the  instructor 
controlling  the  entire  learning 
environment .  Hardware  training 
systems  can  be  designed  to  address 
the  stimulus,  response,  feedback, 
and  next  activity  aspects  of  the 
training  environment.  They  can  be 
designed  in  such  a  way  as  to  be 
transparent  to  the  student  and  still 
provide  the  required  level  of  con¬ 
trol  over  the  training  situation. 
When  the  analyst  enters  into  in¬ 
structional  features  analysis  the 
system  will  scan  all  existing  data 
for  those  tasks  that  drive  hardware. 
Those  tasks  are  then  entered  by  the 
system  into  the  instructional  fea¬ 
tures  analysis  database.  The  system 
brings  each  task  up  and  asks  the 
analyst  to  enter  the  appropriate 
task  taxonomy.  The  system  then  asks 
a  series  of  questions  on  each  of  the 
four  learning  aspects.  The  system 
makes  instructional  features  recom¬ 
mendations  based  on  the  analyst's 
input.  The  system  will  generate  an 
instructional  features  report  that 
can  be  included  into  the  training 
equipment  functional  specification. 

CDRRKHT  APPLICATIOWS 

ISDA  has  replaced  the  "paper"  meth¬ 
odology.  It  is  currently  being  used 
on  the  Joint  Surveillance  Target 
Attack  Radar  System  (Joint  STARS) 
and  B-2  programs .  Joint  STARS  was 
the  first  program  to  use  ISDA  for  a 
weapon  system  training  analysis. 

Many  "growing  pains"  have  been 
worked  out  during  this  implementa¬ 
tion.  In  addition,  the  B-2  program 
is  using  the  system  to  determine 


system  training  requirements  for  3 
level  (semi-skilled)  students.  They 
are  also  collecting  data  to  be  used 
in  a  time  savings  study  on  this 
progreun  versus  the  "paper"  methodol¬ 
ogy. 

The  decision  logic  used  in  ISDA  has 
also  been  incorporated  into  the 
Joint  Service  Logistic  support 
Analysis  Record/Instructional  Sys¬ 
tems  Development  Decision  Support 
System  .  This  is  a  project  spon¬ 
sored  by  the  Armstrong  Laboratory 
aimed  at  automating  the  Logistics 
Support  Analysis  (LSA)  data  flow 
into  the  ISD  process . 

FUTURg  DKVKLOPinMT*; 

We  are  in  the  initial  stages  of 
developing  algorithms  that  will  aid 
in  the  selection  of  instructional 
strategies.  Analysts  are  biased  by 
what  their  experience  has  given 
them.  In  the  traditional  technical 
training  environment,  lecture/dis- 
cussion  and  demonstration/perform¬ 
ance  have  been  the  strategy  of 
choice.  The  development  of  new  media 
and  methods  of  information  transmis¬ 
sion  have  broadened  the  scope  of 
viable  instructional  strategies. 

The  ultimate  goal  is  to  have  the 
system  query  the  analyst  on  areas 
such  as  target  audience  characteris¬ 
tics,  type  of  material  being  taught, 
and  potential  learning  problems . 

The  system  would  then  recommend  a 
specific  strategy  based  on  the 
answers  to  the  questions .  In  addi¬ 
tion,  we  are  working  on  processes 
that  will  aid  in  determining  the 
instructional  features  requirements 
for  CBT.  We  are  focusing  our  ef¬ 
forts  on  a  process  that  will  aid  in 
determining  what  features  are  re¬ 
quired  to  address  individual  learn¬ 
ing  differences,  what  types  of 
remediation  are  required,  branching 
requirements,  etc.  We  envision  a 
system  that  will  query  the  analyst 
(as  done  in  CBT  fidelity  analysis) 
and  ultimately  recommend  the  CBT 
training  strategy,  branching,  reme- 
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diation,  and  feedback  routes  that 
would  best  serve  the  student. 

COWCLOSIOil 

Instructional  Systems  Development 
Automation  is  proving  itself  to  be  a 
viable  tool  to  aid  an  analyst  in 
determining  system  maintenance 
training  requirements.  It  greatly 
reduces  the  amount  of  administrative 
work  that  was  put  into  the  tradi¬ 
tional  analysis.  Just  as  important, 
it  standardizes  the  application  of 
the  analysis  parameters  and  decision 
logic.  The  analysis  is  more  objec¬ 
tive  and  a  better,  more  consistent 
set  of  training  recommendations  is 
the  result.  I  have  seen  the  future 
and  its  name  is  automation,  probably 
sums  up  what  ISDA  is  all  about.  We 
view  the  computer  as  a  tool  to  aid 
the  analyst  in  getting  the  job  done. 
The  more  efficient  we  can  make  the 
tool  and  the  processes  that  it 
interfaces  with,  the  better  the 
return  on  investment.  We  are  not 
quite  to  the  point  where  the  system 
will  ask  the  analyst  to  load  the  LSA 
data,  press  FIO,  and  wait  for  the 
list  of  training  requirements  to 
come  out,  but  give  us  some  time. 
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ABSTRACT 

The  Advanced  Instructional  Design  Advisor  (AIDA)  pro¬ 
ject  is  an  exploratory  research  and  development  effort 
at  the  Air  Force  Armstrong  Laboratory's  Human  Resources 
Directorate  (AL/HRTC) .  The  purpose  of  AIDA  is  to 
establish  the  foundations  for  a  set  of  intelligent  and 
automated  tools  to  assist  military  trainers  in 
designing  and  developing  effective  computer-based  in¬ 
struction  (CBI) .  The  typical  Air  Force  CBI  developer 
is  a  subject  matter  expert  with  limited  training  in  the 
use  of  computer-based  technologies.  Aggravating  this 
lack  of  CBI  design  expertise  is  limited  knowledge  about 
how  to  optimize  the  use  of  CBI  in  various  training 
settings. 

AIDA  is  being  designed  to  incorporate  as  much  course¬ 
ware  design  expertise  as  possible.  The  most  promising 
technology  available  for  this  purpose  involves  the  use 
of  intelligent  lesson  templates,  which  have  pre- 
established  default  values  so  that  they  are  executable 
with  minimal  content  input  from  a  subject  matter 
expert.  This  paper  presents  a  detailed  elaboration  of 
intelligent  templates  and  how  they  will  be  incorporated 
into  AIDA.  The  discussion  includes  a  description  of 
several  such  templates  and  reports  the  results  of  the 
initial  evaluation  of  two  of  these  templates  at  the  Air 
Force  Academy  and  at  the  Lowry  AFB  Technical  Training 
Center.  In  addition,  we  discuss  the  human  factors 
issues  involved  in  presenting  design  guidance  to 
subject  matter  experts  who  have  relatively  little 
experience  in  instructional  technology. 


INTRODUCTION 

Instructional  design  as  a 
discipline  has  existed  for  at 
least  forty  years  (Merrill, 
1971) .  The  traditional 
definition  of  instructional 
design  is  that  it  is  the 
direct  and  practical 
application  of  knowledge  about 
learning  processes  and  tasks 


(Gagne,  1985) .  Over  the  years 
various  learning  theories  and 
task  analysis  methodologies 
have  provided  the  fuel  for  a 
number  of  instructional  theo¬ 
ries.  At  a  very  general  level 
these  various  instructional 
theories  share  this  common 
assumption:  If  you  want  to 

design  effective  instruction, 
then  your  instructional  plans 
should  take  into  account  in- 
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formation  about  the  tasks 
being  trained  as  well  as  in¬ 
formation  about  how  students 
learn  to  perform  a  particular 
kind  of  task. 

As  with  many  disciplines,  it 
is  generally  true  in  instruc¬ 
tional  design  that  the  more 
time  spent  planning  the  less 
time  spent  implementing.  This 
maxim  has  been  largely  ignor¬ 
ed,  however.  When  instruction 
is  delivered  by  human  instruc¬ 
tors  in  a  classroom  setting, 
lack  of  carefully  planned 
instruction  can  be  compensated 
for  by  creative  instructors. 
When  information  is  not  pre¬ 
sented  clearly  to  students, 
instructors  are  confronted 
with  puzzled  faces  and  con¬ 
fused  responses.  Experienced 
instructors  can  then  present 
clarifications,  answer  ques¬ 
tions,  provide  additional 
information,  and  so  on,  in  a 
manner  appropriate  to  the 
situation. 

Using  computers  in  educational 
or  training  situations  com¬ 
plicates  the  process  of  in¬ 
structional  design.  Computer 
software  that  is  designed  for 
instructional  purposes  is 
called  courseware.  Courseware 
that  is  not  carefully  designed 
and  developed  is  likely  to  be 
both  expensive  and  ineffective 
(Jonassen,  1988) .  Most  cours¬ 
eware  is  delivered  in  settings 
without  dedicated  and  experi¬ 
enced  instructors  present  to 
react  to  student  queries  and 
problems.  As  a  consequence, 
the  need  for  very  carefully 
planned  instruction  increases 
as  more  and  more  use  is  made 
of  computer-based  instruction 
(CBI) . 

More  and  more  use  of  CBI 
should  and  will  occur  in  the 


Air  Force.  Courseware  can  be 
used  to  raise  the  quality  of 
training  and  education  by 
providing  review  tutorials, 
drill  and  practice,  meaningful 
simulations,  and  exploratory 
learning  environments,  all  of 
which  have  proven  learning 
value  but  might  be  too  costly 
or  labor  intensive  to  provide 
by  traditional  means.  In 
order  to  make  effective  use  of 
CBI,  it  will  be  necessary  to 
devise  effective  instructional 
design  strategies  appropriate 
for  advanced  computer-based 
insrructional  delivery  set¬ 
tings. 


THE  PROBLEM  ADDRESSED  BY  AIDA 

The  Air  Force  is  confronting 
this  problematic  situation: 

1)  CBI  is  desirable  for  a 
variety  of  reasons  in  a  vari¬ 
ety  of  settings;  2)  CBI  can  be 
expensive  to  produce  —  esti¬ 
mates  range  from  $10,000  per 
hour  of  CBI  and  up  (Lippert, 
1989) ;  3)  There  is  a  scarcity 
of  expert  knowledge  about  how 
to  make  optimal  use  of  CBI. 
This  situation  is  aggravated 
by  the  rapid  development  of 
advanced  interactive  t  ichnolo- 
gies  that  have  applications  in 
CBI  (e.g.,  DVI,  a  form  of 
interactive  digitized  video) 
and  by  ever  decreasing  defense 
budgets . 

AIDA  is  focusing  on  the  design 
and  development  process  (Spec- 
tor,  1990) .  That  process  is 
notoriously  time-consuming 
(Faiola,  1989) .  In  addition, 
efficient  development  is  ham¬ 
pered  by  the  use  of  current 
authoring  systems  which  re¬ 
quire  teams  of  experts  and 
extensive  testing  and  refine¬ 
ment  in  the  development  pro¬ 
cess  (MacKnight  &  Balagopalan, 
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1989) .  In  short,  the  Air 
Force  needs  more  CBI  (CBI  is 
construed  broadly  to  include 
all  forms  of  computer-assisted 
instruction  or  CAI) ,  but  the 
Air  Force  cannot  afford  the 
expense  of  contracting  out  the 
development  of  CBI  and  lacks 
the  expertise  to  produce  ef¬ 
fective  CBI  on  a  large  scale 
basis. 


THE  PROPOSED  SOLUTION 

AIDA  is  intended  to  provide  an 
automated  and  intelligent  set 
of  tools  to  assist  subject 
matter  experts  in  the  creation 
of  effective  CBI  (Muraida  & 
Spector,  1990) .  AIDA  will  be 
a  courseware  authoring 
environment  which  advises  and 
guides  the  user  through  the 
process  of  designing  and 
developing  CBI.  The  basic 
philosophy  is  to  build  into 
AIDA  as  much  CBI  design 
expertise  as  possible  in  order 
to  enable  subject  matter 
experts  with  little  background 
in  instructional  technology  to 
develop  effective  CBI.  The 
intent  is  to  minimize  the  need 
for  system  specialists  in  the 
design  and  development 
process.  AIDA  will  automate 
as  much  of  this  process  as 
possible. 

To  accomplish  this  objective, 
AIDA  will  take  established 
theories  of  knowledge, 
learning,  and  instruction  and 
incorporate  those  theories 
into  a  meaningful  course 
authoring  environment.  An 
extensive  review  of  existing 
authoring  environments 
indicates  that  none  attempt  to 
provide  the  kind  of  assistance 
and  guidance  to  be  available 
in  AIDA  (Hickey,  Spector,  & 
Muraida,  1990) . 


INTELLIGENT  LESSON  TEMPLATES 

While  some  CBI  developers  have 
religiously  applied  tradition¬ 
al  instructional  systems  de¬ 
sign  (ISD)  methodologies  to 
the  design  and  development  of 
courseware,  others  have  con¬ 
cluded  that  radically  improved 
methodologies  are  needed  to 
guide  the  process  of  creating 
consistently  effective  CBI. 
Merrill  and  his  colleagues  at 
Utah  State  University  have 
been  developing  transaction 
theory  as  part  of  a  so-called 
second  generation  of  instruc¬ 
tional  design  (Merrill,  Li,  & 
Jones,  1990) . 

Merrill's  transaction  theory 
is  based  on  the  premise  that 
CBI  should  be  constructed 
around  integrated  computer- 
student  interactions  which 
involve  all  .of  the  activities 
required  to  promote  acquisi¬ 
tion  of  mental  models  appro¬ 
priate  to  mastering  the  task 
or  subject  of  the  instruction. 
Merrill  and  his  colleagues 
have  developed  several  demon¬ 
stration  transaction  shells  to 
illustrate  the  theory.  Two  of 
these  have  been  loaned  to  the 
Armstrong  Laboratory  for  pur¬ 
poses  of  evaluation:  1)  Nam¬ 
ing  the  parts,  and  2)  Check¬ 
list  procedure. 

These  shells  present  the  user 
with  an  interface  that  prompts 
for  input  of  instructional 
content.  The  shell  then  con¬ 
structs  an  executable  student 
module  using  pre-established 
default  values  for  relevant 
instructional  parameters  (e.- 
g.,  number  of  items  to  include 
in  a  test,  passing  score, 
kinds  of  student  interactions 
to  allow,  etc.).  The  user  can 
alter  the  pre-established 
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default  values  and  is  likely 
to  do  so  as  experience  with 
the  system  accrues. 

As  a  consequence,  it  is  fair 
to  describe  these  transaction 
shells  as  intelligent  lesson 
templates  —  in  a  sense,  a 
shell  "knows”  how  its  kind  of 
instruction  should  be  present¬ 
ed  to  students.  AIDA  will 
incorporate  such  templates  and 
an  intelligent  front-end  advi¬ 
sor  to  assist  the  user  in 
determining  which  shells  are 
best  suited  for  a  particular 
instructional  purpose. 


THE  INITIAL  EVALUATION 

AL/HRTC  performed  the  initial 
evaluation  of  the  Naming  Tran¬ 
saction  Shell  at  the  Air  Force 
Academy  in  August,  1990  (Can- 
field  &  Spector,  1991) .  The 
purpose  of  this  pilot  study 
was  twofold;  1)  Determine  how 
to  conduct  future  evaluations 
of  transaction  shells,  and  2) 
Make  a  preliminary  assessment 
of  the  usability,  productivi¬ 
ty,  accessibility,  and  genera- 
lizability  of  the  Naming  Tran¬ 
saction  Shell. 

A  single  subject  was  selected 
for  the  purpose  of  this  study. 
He  had  only  limited  experience 
with  computers  —  knowledge  of 
a  word  processor.  He  was  an 
experienced  instructor  and  had 
developed  the  course  syllabus 
for  the  classroom-based  course 
that  was  chosen  for  conversion 
to  CBI.  The  course  module  to 
be  developed  concerned  the 
names,  functions,  and  loca¬ 
tions  of  125  instruments  in 
the  T-37  cockpit. 


Shell  (see  Figure  1) .  He 
spent  about  12  hours  planning 
the  design  and  development  of 
the  course  module.  The  sub¬ 
ject  was  not  responsible  for 
creating  graphical  materials, 
but  his  planning  time  does 
include  planning  which  graph¬ 
ics  to  include  and  what  should 
be  in  the  graphics.  The  sub¬ 
ject  spent  about  14  hours  on¬ 
line  with  the  Naming  Trans¬ 
action  Shell  authoring,  test¬ 
ing,  and  refining  the  module. 
In  other  words,  the  subject 
spent  about  half  of  the  devel¬ 
opment  on-line  with  the  autho¬ 
ring  system  and  about  half 
off-line  in  a  planning  mode. 

The  net  result  of  this  31  hour 
effort  was  a  fairly  sophisti¬ 
cated  lesson  composed  of  10 
separate  modules  nested  in  3 
layers  using  20  individual 
picture  files  (see  Figure  1) . 
Student  data  is  currently 
being  collected,  but  it  is 
expected  that  it  will  require 
in  excess  of  3  hours  of  stu¬ 
dent  time  to  complete  the 
lesson. 


Figure  1.  Development  Hours 


On-iine  Authoring 

14  25 


The  subject  was  given  about  5 
hours  of  instruction  on  the 
use  of  the  Naming  Transaction 
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This  means  that  the  subject's 
development  time  to  instruc¬ 
tion  time  ratio  was  31:3  or 
close  to  10:1.  A  comparable 
development  to  instruction 
ratio  using  a  traditional 
authoring  tool  (omitting  time 
to  develop  graphics)  would  be 
about  200:1  (Lippert,  1989). 

A  follow-on  study  was  conduct¬ 
ed  at  Lowry  AFB  with  eight 
subjects,  all  of  whom  were 
instructors  at  the  Technical 
Training  Center.  Similar 
results  were  obtained.  These 
subjects  had  little  or  no 
experience  with  CBI  but  were 
able  to  create  meaningful  and 
effective  CBI  lesson  modules 
in  just  30  hours.  Student 
data  was  collected  on  one  of 
the  lessons  to  verify  that  the 
lessons  were  indeed  effective. 
Additional  evaluations  with 
additional  students  and  in¬ 
structors  are  planned. 

In  short,  the  early  indication 
is  that  using  intelligent 
lesson  templates  or  transac¬ 
tion  shells  is  an  extremely 
promising  technology.  While 
these  situations  were  signifi¬ 
cantly  constrained  (e.g., 
rather  simple  but  typical 
technical  subject  matter) ,  two 
immediate  conclusions  are 
possible:  1)  The  transaction 

environment  was  accessible  and 
easily  learned  by  novice  de¬ 
signers,  and  2)  Development 
time  can  be  significantly 
reduced  in  many  typical  situa¬ 
tions  without  sacrificing 
quality  of  CBI. 


THE  ARCHITECTURE  OF  AIDA 

The  implications  of  these 
studies  for  AIDA  is  signifi¬ 
cant.  Based  on  the  positive 
results  just  reported,  it 


appears  that  AIDA  should  in¬ 
corporate  intelligent  lesson 
templates  into  its  architec¬ 
tural  design.  The  current 
plan  is  to  construct  an  AIDA 
prototype  that  contains  a  half 
dozen  intelligent  lesson  tem¬ 
plates.  A  minimal  set  proba¬ 
bly  includes  the  following: 


1.  Naming  the  parts 
(graphically  based) 

2.  Checklist  procedures 
(graphically  based) 

3 .  Teaching  terminology 
(non-graphically  based) 

4.  Classification/decis¬ 
ion  making  (graphically  based) 

5.  Building  a  scenario 
(video  or  animation  based) 

6.  Simulation  of  a  set¬ 
ting  (video  or  animation 
based) 


There  are  obviously  other 
possible  templates  that  could 
be  included  in  the  initial 
system  (e.g.,  a  template  for 
generating  practice  problems) . 
Whether  less  or  more  will 
actually  be  included  in  the 
first  AIDA  prototype  will  most 
likely  depend  on  how  much 
money  is  devoted  to  the  next 
phase  of  the  project.  Each  of 
these  intelligent  templates 
will  be  constructed  so  that  it 
provided  the  following  for  the 
user: 

1.  An  intuitive 
interface  to  allow 
easy  input  of  in¬ 
structional  materi¬ 
als. 

2.  A  set  of  pre- 
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established  instruc¬ 
tional  parameters 
that  are  generally 
appropriate  for  the 
type  of  material 
normally  used  in 
that  template. 

3 .  A  method  for 
easy  alteration  of 
some  of  these  param¬ 
eters  so  that  users 
could  customize  the 
delivery  of  selected 
instructional  mate¬ 
rials. 

4.  A  help  feature 
which  provided  on¬ 
line  explanation  of 
a  particular  item. 

5.  A  student  tryout 
mode  to  use  in  test¬ 
ing  the  lesson  at 
various  points  dur¬ 
ing  development. 

6.  A  catalog  of 
existing  and  possi¬ 
bly  relevant  files. 

7 .  A  method  of 
advising  users  which 
template  might  be 
best  suited  to  a 
particular  instruc¬ 
tional  objective. 

8.  A  method  of 
advising  users  how 
they  might  consider 
altering  some  of  the 
instiructional  param¬ 
eters. 

9.  Data  collection 
on  students  and 
instructors.  Stu¬ 
dent  data  would  be 
used  by  instructors 
to  evaluated  effec¬ 
tiveness  of  particu¬ 
lar  lesson  materi¬ 


als.  Instructor 
data  would  be  used 
by  system  designers 
to  refine  future 
versions  of  AIDA. 

The  overall  architecture  of 
AIDA  based  on  the  use  of  in¬ 
telligent  lesson  templates  is 
depicted  in  Figure  2 . 


Figure  2.  AIDA  Architecture 


CONCIiOSION 

AIDA  promises  to  provide  a 
productive  CBI  authoring  envi¬ 
ronment  that  is  accessible  to 
those  subject  matter  experts 
who  have  had  virtually  no 
previous  experience  with  com¬ 
puters  or  with  CBI.  Merrill's 
transaction  shells  indicate 
that  the  use  of  intelligent 
lesson  templates  is  an  ex¬ 
tremely  promising  technology 
with  the  potential  to  signif¬ 
icantly  reduce  development 
times. 
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ABSTRACT 

In  the  Computer-Based  Training  (CBT)  development  process,  creating  flowcharts  and  storyboards  is  both 
necessary  and  time  consuming.  On  average,  40-50  percent  of  the  total  program  development  time  is  attributable 
to  these  two  activities.  However,  when  developing  IVD,  the  courseware  developer  must  also  track  video 
requirements  specified  on  the  storyboards.  In  high  density,  stiU  frame  applications,  video  requirements  typically 
range  from  15,000  to  100,000  separate  video  shots  or  sequences.  The  use  of  a  database  is  essential  for  this 
process.  In  the  past,  courseware  developers  have,  of  necessity,  manually  generated  storyboards  from  flowcharts 
and  then  manually  inputted  video  requirements  into  the  database.  This  procedure  tended  to  be  both  time 
consuming  and  mistake  prone.  The  Interactive  Courseware  Development  Section  of  the  3300  Technical 
Training  Wing  has  developed  an  automatic  storyboard/database  integration  package  which  allows  the 
courseware  developer  to  electronically  storybo2U'd  using  WordPerfect  and  then  integrate  the  data  into  dBase 
IV.  This  automated  procedure  has  produced  a  significant  decrease  in  development  time,  while  reducing  the 
human  error  factor  when  manipulating  a  massive  amount  of  data. 
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ELECTRONIC  STORYBOARD/DATABASE  INTEGRATION; 

A  KEY  TO  ACCELERATING 
THE  IVD  DEVELOPMENT  PROCESS 


INTRODUCTION 

Interactive  videodisc  (IVD)  technology  is  now 
commonly  being  used  in  both  military  and  industrial 
training.  However,  a  common  complaint  of  IVD  is  the 
development  process  is  too  lengthy  and  too  expensive 
to  produce  a  cost  effective  training  media.  A  major 
factor  in  the  length  and  expense  of  the  development 
process  is  that  the  work  of  the  instructional  designer, 
subject  matter  expert,  courseware  developer,  video 
producer,  and  computer  programmer  must  all  be 
combined  to  create  a  videodisc.  Streamlining  this 
data  collection  process,  while  being  careful  not  to 
omit  any  step,  will  decrease  the  overall  development 
time,  and  in  turn  reduce  project  costs.  This  paper  will 
examine  an  electronic  storyboard,  developed  using 
off-the-shelf  application  software  combined  with  in- 
house  customized  software.  This  integrated  package 
provides  a  detailed  means  of  combining  all  te2un  mem¬ 
ber  inputs  to  the  videodisc,  while  ensuring  all  steps  in 
the  development  process  are  followed. 

Solutions  to  the  storyboarding  problems  are  grad¬ 
ually  becoming  available  in  the  form  of  computer 
software  -  especially  integrated  packages  -  that  allow 
for  the  streamlining  of  the  development  process. 
However,  for  our  environment,  we  needed  a  com¬ 
prehensive  electronic  storyboard  capable  of  being 
modified  as  various  projects  evolve.  To  this  end, 
WordPerfect^  was  selected  as  the  primary  story¬ 
boarding  tool  because  of  its  flexibility  and  capacity  for 
developing  and  using  macros  (a  group  of  user-defined 
instructions  that  can  be  activated  together  when 
needed^).  To  manipulate  the  data,  a  Database  Man¬ 
agement  Information  System  was  needed.  dBase  IV^ 
was  selected  for  this  task.  A  major  obstacle  in  devel¬ 
oping  the  integrated  storyboard  package  was  the  por¬ 
tability  of  the  data  from  the  WordPerfect  format  into 
dBase  IV.  Members  of  the  3302  Technical  Training 
Squadron  (System  Support  Activity(SSA))  developed 
a  customized  software  program  that  will  accomplish 
this  data  transfer.  Through  a  series  of  menus, 
courseware  developers  can  readily  obtain  access,  use 
the  integrated  package,  and  store  all  necessary  infor¬ 
mation  pertaining  to  each  project. 


WORDFEKFECr 

Aside  from  the  comprehensive  wordprocessing  ca¬ 
pabilities  of  WordPerfect,  this  software  package  pro¬ 
vides  a  macro  feature  that  includes  the  flexibility  of  a 
programming  language.  Using  the  macro  editor,  we 
were  able  to  create  various  macros  and  nested  macros, 
that  not  only  automate  the  storyboard,  but  provide 
prompts  and  help  routines  that  an  inexperienced  user 
could  easily  use. 

This  automation  provides  multiple  advantages  in 
the  IVD  development  process:  firstly,  it  eliminates 
the  need  for  a  paper-based  storyboarding  system. 
Besides  saving  paper  and  storage  space  that  would  be 
required  for  a  paper-based  system,  all  project  data  is 
maintained  by  the  computer  and  easily  located  or 
changed  as  the  project  evolves.  Secondly,  all  story¬ 
board  development  efforts  become  standardized.  No 
longer  are  various  team  members  using  storyboard 
conventions  that  are  not  standardized  within  the  proj¬ 
ect  team.  Thirdly,  all  members  of  the  development 
team  are  able  to  utilize  the  same  database  to  manipu¬ 
late  the  data  at  will.  For  example,  items  such  as 
shotlists,  scripts,  graphic  lists,  program  text,  and  pro¬ 
gramming  instructions  are  produced  with  little  effort. 
Fourthly,  the  development  process  time  has  been 
shortened.  The  developer  can  copy  blocks  of  story¬ 
boards  and  use  the  append  or  global  replace  functions 
of  WordPerfect  to  make  changes.  Finally,  and  most 
importantly,  since  the  automated  storyboard  is  an 
integrated  package,  not  a  single  storyboard  is  inadver¬ 
tently  omitted  from  the  database. 

Another  feature  of  WordPerfect  that  was  used  to 
enhance  the  electronic  storyboard  was  the  capability 
of  changing  the  keyboard  definition.  By  reassigning 
the  purpose  of  specific  function  keys  (i.e.,  FIO,  Shift- 
FIO,  Control-FlO)  we  designed  a  simple  to  use  human 
interface  that  an  inexperienced  user  can  use.  With  the 
newly  defmed  function  keys,  the  new  developer  can 
activate,  and  utilize  the  electronic  storyboard  with 
minimum  training. 

Specific  fields  of  the  storyboard  (Figure  1.),  which  are 
activated  by  a  macro,  are  provided  to  accommodate  the 
various  team  members  assigned  to  the  development 
team.  Even  though  this  storyboard  is  generic,  the  macro 
can  easily  be  modified  to  accommodate  any  particulars 
(i.e.,  special  branching  instructions,  or  special  menu 
items,  etc.)  of  a  specific  project. 
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STORYBOARD  # 

The  first  field  the  developer  will  encounter  deals 
with  the  numbering  scheme  of  the  storyboard.  The 
developer  inputs  a  munbering  scheme  which  be¬ 
comes  the  storyboard  number.  It  is  important  that 
each  developer  be  consistent  in  the  numbering  se¬ 
quence,  as  ^  future  work  will  be  accessed  by  this 
number.  As  a  local  convention,  this  storyboard 
number  can  be  traced  directly  back  to  the  project 
flowchart(s). 

SMPTETIME: 

Initially,  this  field  is  left  blank  since  the  SMPTE 
(Society  of  Motion  Picture  and  Tblevision  Engi¬ 
neers)  time  code  is  not  available  until  later  in  the 
development  process.  When  the  SMPTE  time  code 
is  available  (diuing  the  video  shoot),  the  developer 
enters  the  required  information.  Once  entered,  the 
macro  program  automatically  converts  the  SMPTE 
time  code  to  the  appropriate  frame  number.  This  is 
especially  important  during  the  programming  stage, 
since  coding  of  the  videodisc  can  only  be  accom¬ 
plished  through  the  various  ft'ame  numbers. 

FRAME  NUMBER: 

This  field  displays  the  converted  frame  number 
once  the  SMPTE  time  code  is  entered  in  the  previ¬ 
ous  field. 

NARRATION: 

This  field  of  the  storyboard  allows  the  developer 
the  opportunity  to  enter  the  narration  or  audio  that 
will  be  used  on  the  particular  storyboard.  As  with 
all  other  fields,  if  there  is  no  entry  needed,  it  is  left 
blank. 

AUDIO  TRACK  1: 

This  field  is  simply  a  Yes/No  response  as  to 
whether  audio  track  1  is  to  be  used. 

AUDIO  TRACK  2: 

This  field  is  simply  a  Yes/No  response  as  to 
whether  audio  track  2  is  to  be  used. 

SCREEN  TYPE: 

The  developer  identifies  the  screen  type  (a  stan¬ 
dard  convention  developed  within  the  organization 
that  identifies  a  screen  template  to  be  used  when 
frammg  a  video  shot)  that  will  be  employed  with  this 
particular  storyboard. 

AUTHOR: 

Identifies  the  courseware  developer  responsible 
for  the  storyboard. 

TASK  NUMBER: 

Storyboard  reference  to  particular  task  item. 


TECH  ORDER  REFERENCE: 

This  field  is  used  to  identify  the  subject  technical 
reference  of  the  storyboard. 

SPLITSCREEN: 

A  Yes/No  prompt  that  identifies  whether  the 
video  presented  on  the  monitor  will  be  in  a  split 
screen  format. 

If  the  response  to  this  field  is  “Yes”,  the  story¬ 
board  generates  multiple  object  descriptions,  other¬ 
wise  it  only  presents  a  single  subject  description. 

GRID/LOCATION: 

A  reference  (either  a  grid  location,  specific  loca¬ 
tion,  or  particular  subject  matter  of  the  video  shot) 
to  distinguish  the  position  of  the  subject  matter  that 
must  be  photographed. 

NOMENCLATURE: 

The  specific  name  of  the  subject  of  the  video  shot. 

DESCRIPTION: 

A  narrative  description  of  the  required  shot. 

TYPE  SHOT: 

A  two  letter  identifier  (CU-Close  Up,  MV-Me- 
dium  \fiew,  FV-Full  View,  and  MO-Motion  Se¬ 
quence)  that  specifies  the  type  of  video  shot. 

VIDEO  TEXT: 

In  this  field  the  developer  enters  any  text  that 
requires  chyron  generation  (video  text  displayed 
directly  from  the  videodisc). 

COMPUTER  TEXT: 

Ibxt  that  will  be  generated  by  the  authoring  sys¬ 
tem. 

BRANCH  TO  STORYBOARD  #: 

Identifies  the  type  of  branch  and  the  frame  the 
student  is  to  be  branched  to.  The  developer  is 
prompted  as  to  what  type  of  branch,  destination 
frame,  and  whether  or  not  multiple  branches  will  be 
used. 

SPECIAL  BRANCH  INSTRUCTIONS: 

The  developer  may  identify  in  this  field  any  spe¬ 
cial  programming  requirements  needed  for  the 
frame  or  any  type  of  special  branching  instructions. 

REVISION  DATE: 

The  macro  automatically  displays  the  revision 
date  of  the  storyboard. 

In  addition,  the  courseware  developer  may  dis¬ 
play  a  graphic  representation  of  the  subject  matter 
on  the  storyboard  (within  the  figure  box).  This 
could  be  accomplished  automatically  by  scanning 
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the  subject  matter,  and  using  the  graphic  capabilities 
of  WordPerfect. 


STORYBOARD#:  | 

SMPTETIME:  | 

FRAME  NUMBER:  | 

NARRATION;  |  - 

AUDIO  TRACK  1:  | 

AUDIO  TRACK  2:  | 

SCREEN  TYPE;  | 

AUTHOR:  | 

TASK  NUMBER:  | 

TECH  ORDER  REFERENCE:  | 

SPOT  SCREEN:  NO  | 

GRID/LOCATION:  | 

NOMENCLATURE:  | 

DESCRIPTION;  | 

TYPE  SHOT:  | 

VIDEO  TEXT:  | 

COMPUTER  TEXT;  | 

UBR  BRANCH  TO  STORYBOARD  #:  | 
END  BRANCH 

SPECIAL  BRANCH  INSTRUCTIONS:  | 
REVISION  DATE:  | 


Figure  1.  -  Electronic  Storyboard 


PORT  TO  dBASE  IV 

After  completing  all  inputs  to  the  electronic  story¬ 
board  and  saving  the  data,  the  developer  must  convert 
the  saved  file  from  WordPerfect  format  to  an  ASCII 
(American  Standard  Code  for  Information  Inter¬ 
change)  format.  This  conversion  is  required  since  the 
storyboard  is  initially  saved  in  a  WordPerfect  format, 
£uid  dBase  IV  does  not  recognize  WordPerfect  com¬ 
mands.  The  conversion  is  accomplished  using  the 
WordPerfect  conversion  program.  The  convert  pro¬ 
gram  allows  conversion  from  selected  programs  to  the 
WordPerfect  format  or  from  WordPerfect  format  to 
other  text  forms.  Upon  activating  the  conversion  pro¬ 
gram,  the  developer  is  prompted  to  provide  the  input 
file  name  and  the  output  file  name  that  the  conversion 
program  will  write  to.  The  output  file  name  must  be 
different  from  the  input  file  name.  Once  the  output 
file  is  identified  and  the  conversion  is  complete,  the 
ASCII  file  must  be  ported  to  dBase  IV. 

The  delimiting  program  was  written  in  Micro.soft  C 
5.0'*.  The  purpose  of  this  program  is  to  generate  the 


delimited  file  for  importation  into  dBase  IV.  Held 
names  in  the  database  system  correspond  to  the  topic 
names  (storyboard  fields)  in  the  WordPerfect  file. 
Therefore,  the  delimiting  program  can  search  the 
ASCII  file  for  these  topic  names  and  once  found,  cross 
reference  them  with  the  dBase  IV  field  names.  When 
a  match  is  found,  the  delimit  program  extracts  the 
information  from  the  ASCII  file  and  exports  it  to  a  file 
compatible  with  the  dBase  IV  field.  Selecting  this 
format  for  the  importation  has  increased  the  portabil¬ 
ity  of  the  storyboard  data.  This  type  of  delimitation  is 
commonly  used  to  support  importing  or  exporting 
into  many  other  database  management  systems. 


dBASE  IV 

appucations  program 

After  the  storyboards  have  been  created  and  saved 
in  WordPerfect,  converted  into  an  ASCII  format  and 
then  delimited  with  the  delimit  program,  it  is  time  to 
put  the  information  into  an  easy  to  use  format.  “Easy 
to  use”  means  that  the  developer  will  be  able  to  access 
the  storyboards  and  data  desired  by  specifying  a 
search  criteria.  For  example,  during  a  video  shoot,  the 
video  producer/director  can  reduce  video  production 
time  by  reducing  camera  repositioning.  To  accom¬ 
plish  this,  the  shot-list  must  be  able  to  group  subject 
matter  and  camera  angles.  Therefore,  by  specifying  a 
search  criteria  for  all  storyboards  that  use  a  specific 
subject  matter  and  camera  angle,  the  specified  shot- 
list  can  be  produced. 

The  package  that  was  decided  upon  that  would  give 
us  the  data  in  the  format  desired  was  dBase  IV.  Be¬ 
sides  having  the  ability  to  store  and  retrieve  informa¬ 
tion  in  the  format  desired,  dBase  IV  also  had  the 
added  feature  of  a  powerful  applications  generator. 
An  appUcations  generator  is  a  tool  used  in  automatic 
code  generation  of  program  files.  We  opted  to  use  the 
dBase  IV  Applications  Generator  to  shorten  the  pro¬ 
gram  development  time.  By  using  the  generator, 
other  features  were  included  in  the  applications  pro¬ 
gram,  such  as  networking.  Even  though  a  network  is 
currently  not  used,  the  ability  is  there  for  future  use. 
One  of  the  disadvantages  of  using  an  applications 
generator  is  that  the  code  generated  suffers  from  what 
is  known  as  “code  bloat”.  In  other  words,  the  resulting 
code  is  not  necessarily  streamlined  cr  the  most  effi¬ 
cient,  however  for  our  purposes,  the  advantages  of  the 
generator  outweighed  the  disadvantages. 

Upon  entering  the  dBase  IV  applications  program, 
each  developer  is  pathed  to  a  menu  screen  (Figure  2.) 
from  where  they  are  able  to  manipulate  data.  The 
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following  is  a  synopsis  of  the  functions  available  from 
the  applications  menu; 


APPEND  WordPerfect  Data 

Add  Info  Manuadly 
Change  Information 
Browse  Information 
Sort  Information 
Generate  Report 
Exit  to  Command  Center 
Exit  to  DOS 

<F1>  for  Help 

Figure  2. 

APPEND  WORDPERFECT  DATA: 

The  append  function  of  dBase  IV  allows  users  to 
add  records  to  the  end  of  an  existing  database  file. 
The  applications  program  takes  the  delimited 
ASCII  file  that  was  created  from  the  delimit  pro¬ 
gram  and  appends  the  information  into  a  database 
file.  This  database  file  is  created  automatically,  and 
is  appended  each  time  new  information  is  added 
into  the  delimited  ASCII  file. 

ADD  INFO  MANUALLY: 

This  option  allows  the  developer  to  add  informa¬ 
tion  manually  to  the  database  file.  Caution  must  be 
taken  using  this  function  smce  the  automatic  infor¬ 
mation  transfer  between  dBase  IV  and  WordPerfect 
is  a  one-way  track.  Information  can  only  be  pro¬ 
cessed  automatically  from  WordPerfect  into  dBase 
IV,  and  not  vice-versa.  If  the  developer  needs  to  add 
information  to  the  storyboard,  it  is  recommended 
that  this  information  be  added  into  WordPerfect, 
and  then  “appended”  into  dBase  FV. 

CHANGE  INFORMATION: 

This  option  allows  the  developer  to  alter  existing 
information  contained  within  the  database.  Again, 
as  with  “Add  Info  Manually”,  care  should  be  taken 
using  this  function  due  to  the  information  transfer 
between  programs.  If  the  informations  is  changed 
manually  within  the  database,  it  must  be  changed 
manually  within  Word  Perfect. 

SORT  INFORMATION: 

This  option  sorts  the  information  contained  in  the 
database  by  the  storyboard  number.  If  there  are 
duplicate  storyboard  numbers,  the  oldest  story¬ 
board  record  is  deleted,  and  then  all  remaining 
records  are  sorted  in  an  ascending  manner.  This 
option  should  be  used  following  any  alteration  or 
manual  addition  of  the  information  contained  within 
the  database. 


GENERATE  REPORT: 

This  option  will  display  a  dBase  FV  menu  contain¬ 
ing  report  forms  that  may  be  generated  by  the  sys¬ 
tem.  Depending  on  the  need,  shot  lists,  graphic 
requirements,  programming  instructions,  lesson 
text,  etc.,  may  be  produced.  Each  project  team  may 
develop  additional  reports  as  the  need  arises. 

EXIT  TO  COMMAND  CENTER: 

This  option  paths  the  developer  to  the  dBase  FV 
“dot”  prompt,  (consists  of  a  single  dot  indicating 
dBase  FV  is  ready  to  receive  a  command,  similar  to 
the  DOS  >  prompt) 

EXIT  TO  DOS: 

This  option  exits  the  dBase  FV  application  pro¬ 
gram  and  returns  the  developer  to  the  DOS  prompt. 


In  the  Computer-Based  Training  (CBT)  develop¬ 
ment  process,  creating  flowcharts  and  storyboards  is 
both  necessary  and  time  consuming.  On  average, 
40-50  percent  of  the  total  program  development  time 
is  attributable  to  these  two  activities.  With  the  addi¬ 
tional  requirement  of  incorporating  video  in  an  FVD 
training  project,  the  development  process  becomes 
even  more  complex  and  time  consuming.  In  the  past, 
courseware  developers  have  of  necessity  manually 
generated  stroyboards  firom  flowcharts  and  then  man¬ 
ually  inputted  video  requirements  into  the  database. 
This  procedure  tended  to  be  both  time  consuming  and 
mistake  prone.  To  facilitate  this  complex  data  collec¬ 
tion  process,  an  automatic  integrated  software  pack¬ 
age  was  created.  This  automated  package  has 
produced  a  significant  decrease  in  development  time, 
while  reducing  the  human  error  factor  when  manipu¬ 
lating  a  massive  amount  of  data.  It  must  be  noted, 
however,  that  this  package,  although  effective,  is  not 
Finalised.  Further  customizing  will  evolve  as  needs  of 
future  projects  arise. 


ENDNOTES 

1.  WordPerfect  is  a  registered  trademark  of  Word 
Perfect  Corporation. 

2.  MimDBK-ICWA,  Interactive  Ctmrseware  Glos¬ 
sary  of  Terms  for  Department  of  Defense  Interac¬ 
tive  Courseware  (ICW)  Acquisition  Manager's 
Guide,  Joint  Service  Action  Group:  11  July  1989. 

3.  dBase  IV  is  a  trademark  of  Ashton-Tate  Corpo¬ 
ration. 

4.  Microsoft  C  S.O  is  a  registered  trademark  of 
Microsoft  Corporation. 
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A  MODI^L  roll  COMIM^rUNC.'Y  HASH!)  riiAINING  ANf)  DEVnLOPMENT 


Donna  Gtcpoiy  and  Ed  Rao 
Ahsiratl 


I\)day’.s  cinplt'ycts  have  a  eliallcnj^e  lo  meet 
Ihc  needs  of  a  rapidly  thanking  work  plaec  and 
work  force.  The  U.S.  General  Services 
Adniinislration  (GSA)  has  formali/ed  ils  training 
siralcpy  for  professional  developmeni  of  ils 
employees,  through  eslahlishmenl  of  an 
Oecupalional  Cerlifiealion  Program.  This  is  a 
eompelcncy-ba.scd  development  program  which  was 
implemented  in  1988  lo  meet  lhe.se  demands. 

litis  paper  describes  the  multipurpose  job 
analysis  procedure  used  lo  develop  a 
compeicney-based  training  model  and  related 
performanee  standards  to  reinforce  training.  I  his 
provides  a  system  for  relating  the  re(|uirenienls  of 
Ihc  job  to  Ihc  abilities  of  Ihc  employees.  It 
focu.ses  on  training  requirements,  training  .sources, 
and  the  proficiency  of  performance. 

DONNA  GREGORY'  is  a  Personnel  Psychologist 
with  Ihc  Office  of  Quality  Management  and 
Training  in  the  U.S.  General  Services 
Administration.  She  is  responsible  for  the 
agency-wide  Oecupalional  Cerlificilion  Program 
which  euirenlly  covets  .1.1  of  GSA's  major 
profc.s.sions,  and  Ihc  management  and  executive 
development  pittgrains.  She  implemented  a 
multipurpose  job  analysis  method  in  CJSA  which 
has  led  lo  Ihc  establishment  of  inlegialed 
personnel  documents  for  pnrprrscs  of  selection, 
performance  evaluation,  and  training  and  career 
development. 

Ill  RAO  is  an  Electrical  fotgincer  with  Ihc  Office 
of  Physical  Security  and  l.aw  Enforcement  in  the 
U.S.  General  Services  Administration.  lie  is 
responsible,  in  a  technical  capacity,  for  security 
systems  policies  and  programs.  Mr.  Rao  came  t«) 
GSA  after  working  fifteen  years  in  various  project 
engineering  positions  in  the  private  sector  and  his 
experience  includes  working  on  Airpoit  security 
.systems  projects  in  Saudi  Arabia. 
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A  MODFIL  I'OR  COMI’BI  fiNCY  BASRD  TRAINING  AND  DEVELOPMENT 


lnIroUnclion 

rorccasis  for  the  fulurc  make  it  clear  ihat  new 
approaches  (o  employee  rccruitriicrK,  dcvcIopmciK, 
and  relenlion  are  necessary.  I  Employers  face 
many  challenges  dues  (o  Ihc  rapid  technological 
and  demographic  changes.  With  these  changes, 
the  idea  of  hiring  fully  trained  workers  is 
disappearing.  Industry  and  public  .service 
organi/irtions  must  develop  their  work  force  to 
meet  their  special  needs. 

The  U.S.  General  Services  Administration  (GSA), 
a  central  management  agency  that  .sets  prrlicy  in 
such  areas  as  govcrrimem  protiirciiicnl,  real 
property  management  and  telecommunications 
services,  began  to  hurk  at  transitional  training  for 
the  1990s  .several  years  ago.  It  is  necessary  to 
identify  more  efficient  ways  to  train  the  svork  force 
and  to  remain  competitive.  To  inrprovc  prrrgram 
performance  and  employee  development,  and  to 
profe.ssionali/c  the  work  force  GSA  established  Ihc 
Occupational  Certification  I’rrrgram.  A 
compcicncy-ba.scd  approach  was  u.sed  to  cslabli..h 
a  certification  training  nuulcl  which  focuses  on 
Irnininp  for  new  skilfs,  tipgrading  skills  and  career 
tlcvclopmcnl.2  lltc  training  model  is  reinforced  by 
performance  standards  that  arc  compcicncy-bascd 
with  measurable,  ob.scrvable,  and  attainable 
outcomes. 

Backcround 

Several  ptddic  and  private  sector  studies  by  the 
Hudson  Inslilule,  Ihc  Rand  Corporation,  and 
others  have  identified  poor  training  as  a  major 
catsc  of  high  turnover  and  poor  performance..! 
Training  does  not  slop  at  Ihc  school  door.  Sixty 
percent  of  an  individual’s  training  occurs  in  Ihc 
work  place.  The  focus  must  be  on  education, 
training,  and  retraining.  The  Certification 
Program  establishes  minimum  professional 
standards  and  Ihc  training  and  development 
rctpiircmcnls  for  the  profession. <1  This  training 
slriilegy  is  in  place  for  !!  occupations,  including 
support  staff,  profc.ssional  and  administrative 
personnel,  as  well  as  for  supervisors,  managers, 
and  executives.  !he  Certification  Program  uses  a 
competency-based  training  iikkIcI.  (Figure  I 
rcpre.scnls  the  competency  model  in  a  flowchart 
fortnai.) 


Mclhodology 

A  muilipur|rosc  job  analysis  method  was  used  to 
provide  Ihc  basis  for  conicni-oricnicd  validation  of 
Ihc  occupational  training  plans  and  the  related 
performance  slandards.5  Tliis  data  also  was  used 
for  developing  selection  procedures.  The  job 
analysis  approach  allows  the  development  of  an 
integrated  system  of  personnel  documents  needed 
In  meet  overall  personnel  management 
responsibilities.  This  paper  will  focus  on  the 
training  model  and  performance  standards. 

GSA’s  Career  Development  and  Training  Division 
ciMrrdinalcd  Ihc  trvcrall  development  of  Ihc 
Certification  Prrrgram.  However,  it  is  cs.scnlial 
that  program  development  and  implementation 
become  a  joint  effort  between  management  and 
Ihc  human  resource  organization.  Management 
must  lake  an  active  role  in  diagnosing  training 
needs,  rwcrscclng  program  development  and 
administration,  and  in  some  eases  even  delivering 
Ihc  lraining.6 

A  job  analysis  was  conducted  for  each  «Kcupaiion. 
Task  groups  consisting  of  j«)b  experts,  typically 
senior  level  employees  trr  pr«>gram  managers,  were 
a.sscmblcd  by  occupation.  The  occupational  job 
experts  generated  a  list  of  work  behaviors  or  job 
duties  which  were  further  broken  down  into  the 
specific  job  tasks  necessary  to  perform  each  duly. 
!1ic  job  experts  evaluated  Ihc  tasks  on  .several 
dimensions,  including  importance  to  Ihc  job  and 
percentage  t)f  time  spent  performing  Ihc  tasks. 
Competencies  needed  for  the  performance  »)f  the 
tasks  were  identified  and  rated  on  overall 
importance  to  Ihc  job,  time  spent  performing, 
necessity  at  entry,  and  extent  the  c«mipclcncy 
differed  among  levels  of  job  performance. 

llic  job  analysis  identified  and  clearly  defined  the 
major  duties,  job  tasks,  and  learning  »>bjcclivcs. 
The  competencies  or  knowicilgc,  skills  an«l  abilities 
needed  to  perform  Ihc  job  tasks  were  defined. 
Training  curriculum  was  then  linked  lr>  each  of  Ihc 
competencies.  The  job  experts  determined  the 
most  appropriate  types  of  training  activities  for  Ihc 
development  or  enhancement  of  each  «rf  the 
competencies.  A  wide  variety  of  format  and 
informal  training  activities  were  identified, 
including  formal  chissroom  training,  rotational 
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assignmciUs,  scniiiiats,  \vutk.sh<»ps.  <icl(  liaininp 
iiistriiciion,  and  on-lhc-joh  Irainiiij;  asst);niucnts 
Divcrsily  «)f  cxpciicntc  and  exposure  lo  diffeieni 
kinds  of  assignnienls  was  einphasi/ed. 

MuKipurpose  Joli  Analysis 

An  inicgralcd  niullipiirposc  job  analysis  approach 
provides  ihc  basis  for  the  cnnicnl  validation 
strategy  used  to  supirort  personnel  processes  such 
as  selecturn,  performance  appraisal,  and  training  ? 
This  method  for  collecting  task-oriented 
information  complies  with  regulatory  guidance 
(Uniform  Guidelines  rrn  employee  Selection 
Procedures)  and  provides  ihc  direct  linkage  and 
job-related  documentation  to  support  the 
development  of  personnel  documents. 
(Government  Employee  Relations  Report 
I77-CV-.1431  829:74  (para  18.^)).  Eurthcr,  ihc 
results  produced  using  multipurpose  job  analysis 
systems  have  a  demonstrated  history  of  acceptance 
to  the  courts.  (U.S.  v.  New  York.  W.G.  Connclic, 
ct  al.,).  Tlic  method  meets  both  the  p.sychomctric 
and  legal  issues  affecting  job  analysis  practices. 
(Demis,  Belenky,  and  Sodcr  (PW). 

The  ability  to  tic  the  job  tasks  and  knowledge, 
skills,  and  abilities  into  an  integrated  personnel 
approach  which  produces  the  selection  criteria, 
training  criteria,  and  performance  evaluatirur 
criteria  incrca.ses  Ihc  power  and  utili/ation  of  Ihc 
method  and  Ihc  producls.8 

riic  Ccriificalioii  Model 

A  training  plan  identifies  Ihc  job  functions, 
crmipclcncics,  and  training  and  development 
needed  for  an  individual  lo  progress  through  Ihc 
profc.ssion,  from  Ihc  entry  level  lo  Ihc  journey 
level.  The  plan  prrwides  a  formal  and  structured 
career  development  path  with  specific  training  and 
development  activities  matched  lo  each 
competency  and  designed  for  employees  to  gain 
cxpcrli.se  in  their  profc.ssion.s. 

Employees  can  be  formally  certified  in  their 
profc-ssions  once  they  have  met  Ihc  full  set  of 
certification  criteria  and  have  demonstrated 
proficiency  of  the  rc(|uircd  competencies.  The 
cm|)loycc's  training  and  progrc.ss  is  monitored  by 
Ihc  su|)crvisor  and  an  Occupational  Review  Panel 
consisting  of  job  experts  in  Ihc  ageiu^.  I  hc  Panel 
recommends  certification  of  eligible  employees  Irr 
lop  management  officials. 


Tlie  major  objectives  of  the  program  arc: 

o  I'o  orient  new  employee  Kr  their  jobs. 

o  To  improve  Ihc  performance  capabilities  of 
employees. 

r>  To  prepare  employees  for  advancement  and 
career  dcvchrpmcnt. 

o  To  improve  employee  morale  and  job 

.satisfaction. 

The  job  analysis  information  provided  the 
foundation  for  Ihc  development  of  the.  formal 
certification  training  plan.  (See  Figure  2 
illustrating  the  linkage  of  managerial  competencies 
to  training  course  curriculum.)  This  process 
relates  job  functions  or  tasks  lo  conditions  of 
training  in  order  lo  .select  optimal  methrxls  and  of 
training  and  measures  of  job  pcTformancc.9 
Training  tasks  arc  established  to  reach  training 
goals. 

An  applicalitrn-oricnicd  approach  lo  training  with 
mcasrrrablc  and  rrbscrvabic  rrulcomcs  is  important 
to  a  successful  development  prrrgram. 
Performance  rrbjcclivcs  cannot  be  ignored  when 
selling  up  a  training  program.  Compelcncy-ba.sed 
performance  standards  were  developed  irsing  Ihc 
job  analysis  information.  This  created  a  model 
performance  plan  with  observable  and  directly 
relevant  job  behaviors.  Tliis  plan  serves  as  a  brrrad 
description  of  Ihc  type  of  performance  that  is 
expected  of  a  typical  employee  in  Ihc  occupation. 
Performance  standards  were  developed  that 
identify  specific  examples  of  behavior  for  cxpccter! 
performance  and  frrr  performance  that  significantly 
exceeds  normal  expectations.  The  linkage  between 
Ihc  training  plan  ami  Ihc  performance  plan  is 
critical  lo  employee  development.  It  prtrvidcs  a 
system  for  relating  Ihc  requirements  of  the  ji>b  lo 
Ihc  abilities  of  Ihc  cmpIoycc.s.  It  creates  a  means 
to  provide  relevant  feedback  lo  employees  on  their 
performance  and  development.  10 

The  Certification  Program  focuses  on  the 
proficiency  of  performance  and  application  in  a 
work  selling.  Competencies  focus  on  performance 
as  opposed  lo  knowledge.  The  training  model  is 
reinforced  by  performance  measures  which 
enhances  Ihc  focus  of  lraining.il  (See  Figure.  .1 
which  illustrates  the.  multipurpose  job  analysis 
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approach  and  ilic  linkage  of  performance 
rccpiircmcnl.s  lo  (raining  and  employee  cvaliiadon.) 

I  licldiulits  of  (lie  rian 

Several  uniipie  features  of  lliis  |)lan  are: 

1.  The  Certification  plan  is  based  on  a 
competency  training  model,  wherein  a  thorough 
job  analysis  identifies  the  (pialifications  or 
competencies  of  a  particular  position. 

2.  Com|)ctcnc7  focuses  on  perforntance  as 
opposed  to  knowledge.  As  in  a  work  setting  the 
application  of  (he  knowledge  is  more  important 
than  simple  recall. 

.3.  Training  (asks  established  to  reach  (raining 
goals,  integrate  work  assignments  with  formal 
(raining  and  special  scminais  in  relevant  topies  of 
job  responsibilities. 

d.  Tlic  individual  development  plan  has  a 
specific  cour.se  of  action,  with  (raining  resources 
idc^iitified  for  ease  rrf  implementation. 

5.  The  individual  development  plan  is  in  a 
checklist  format  (hat  makes  it  easy  for  the 
cmphjycc  and  supervisor  lo  ndminhicr  and 
monitor  (he  progress  of  (he  participant  in  (he 
Certifieation  Plan. 

6.  Competency-based  performance  standards  with 
measurable,  tib.scrvablc  and  attainable  outcomes 
arc  built  into  model  performance  plans.  The 
performance  measures  are  used  to  reinforce 
(raining  and  to  provide  a  method  for  relating  the 
rctpiiremcnls  of  the  job  lo  the  abilities  of  the 
employees. 

Conclusions 

Today’s  (raining  philosophy  needs  lo  rcllcct  our 
current  situation.  It  is  ncces.sary  lo  provide 
incen(ivc.s,  challenges,  and  encouragement  lo  all 
employees  lo  continue  lo  learn  and  grow,  both 
professionally  and  personally.  The  GSA 
Ccrtifioition  Program  is  designed  not  only  lo 
attract  motivated  and  rpialificd  individuals,  but  lo 
challenge  them  with  opportunities  to  expand  (heir 
knowledge  and  stay  abreast  of  new  (rends  in  (heir 
fields.  The  program  ensures  that  our  employees 
have  the  skills  and  nccc.s.sary  training  to  perform 
(heir  current  jobs.  It  provides  the  guidance  for 


cross-training  and  retraining  employees  to  meet 
work  force  needs  for  the  future. 

'Hie  program  promotes  the  deveUrpinent  rrf 
employees  which  it  is  hoped  will  ullimalcly 
enhance  the  quality  of  performance  and 
productivity  of  the  overall  wo.^k  force.  Through 
ccrtifiaiiion,  employees  are  rccogni/cd  for 
professional  competence  in  (heir  professions. 
Employees  completing  (he  program  arc  recognized 
and  receive  a  certificate  of  accomplishment. 
Career  development  is  an  effective  recruitment 
incentive  for  individuals  entering  the  government. 
It  ensures  that  (hey  will  receive  training  to  attain 
the  knowledge  and  skills  needed  for  success  in 
(heir  liclds.  The  program  serves  lo  retain 
employees  by  providing  job  enrichment  and 
opportunities  for  cro.ss  training  or  new  skill 
development.  Tlic  agency  receives  the  benefits  of 
an  efficient,  effective,  and  productive  work  fttrcc. 

Tills  Certification  Program  has  been  immensely 
beneficial  to  the  individual  candidates  through 
(heir  development;  to  management  by  creating  a 
professional  staff;  to  the  agency  by  enhancing  (he 
efficiency  and  productivity  of  the  work  force;  and 
lo  the  customer  by  enhancing  the  quality  and 
timeliness  of  GSA  service  delivery.  An  evaluation 
of  the  program  Is  currently  underway. 
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DEVELOPMENT  OF  A  CONCEPTUAL  METHODOLOGY  FOR  A 
COMPUTER-BASED  AIDING/TRAINING  DECISION  SUPPORT  SYSTEM 


John  P.  Zenyuh,  William  B.  Rouse,  Phillip  C.  Duncan 
Paul  R.  Frey,  and  Theodore  A.  Lamb 

The  ever  increasing  complexity  of  operational  Air  Force  systems  coupled  with 
decreasing  force  levels,  declining  entry-level  skills  and  the  need  to  reduce 
military  expenditures  is  placing  an  unprecedented  demand  on  Manpower,  Personnel 
and  Training  (MPT)  agencies  to  improve  operational  effectiveness  with  fewer 
available  resources.  One  component  of  the  overall  solution  is  the  selection  of 
effective  and  efficient  training/aiding  programs.  However,  selecting  among  the 
wide  variety  of  training  and  aiding  alternatives  (and  the  possible  combinations) 
is  difficult  when  a  myriad  of  interdependent  factors  such  as  system  design  impact 
(e.g.,  to  automate  or  not  automate),  aiding/training  development  costs,  and 
implementation  requirements  must  be  simultaneously  resolved.  In  response  to  this 
need,  the  Air  Force  Human  Resources  Laboratory  at  Brooks  AFB  has  sponsored  the 
Job  Aiding/Training  Allocation  Technologies  (JATAT)  program  for  the  purpose  of 
developing  a  conceptual  aiding/training  decision  support  methodology  for 
personnel  in  complex  systems. 

The  fifteen-step  methodology  developed  during  this  effort  is  a  human-system 
performance-based  approach  to  identifying  applicable  aiding/training  alternatives 
for  complex  system  tasks  and  for  evaluating  the  candidate  aiding/training 
solution  sets.  The  fifteen  steps  can  be  summarized  by  the  following  six 
activities:  Identify  Tasks;  Assess  Human  Limitations,  Abilities,  and 

Preferences;  Determine  Alternatives;  Formulate  Tradeoffs;  Analyze  Tradeoffs;  and 
Integrate  Tradeoffs.  This  paper  describes  the  JATAT  methodology  and  the 
subsequent  data  requirement  issues. 
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DEVELOPMENT  OF  A  CONCEPTUAL  METHODOLOGY  FOR  A 
COMPUTER-BASED  AIDING/TRAINING  DECISION  SUPPORT  SYSTEM 


John  P.  Zenyuh 
William  B.  Rouse 
Phillip  C.  Duncan 
Paul  R.  Frey 
Theodore  A.  Lamb 

INTRODUCTION 


The  ever  increasing  complexity 
of  operational  Air  Force  systems 
continues  to  place  greater  demands  on 
the  personnel  operating  and 
maintaining  them  (AFHRL  Report, 
1986).  The  increased  sophistication 
of  these  systems  coupled  with 
decreased  force  levels,  declining 
entry-level  skills,  and  the  need  to 
limit  military  training  spending  are 
forcing  Manpower,  Personnel,  and 
Training  (MPT)  agencies  to  seek  more 
efficient  methods  of  maintaining  and 
improving  operational  readiness 
(Booher,  1978;  Duncan,  1985). 

In  this  environment  of  "doing 
more  with  less"  the  issues  of 
training  and  job  aiding  are 
paramount.  Technical  training  serves 
as  the  source  of  knowledge  and  skills 
essential  to  task  performance.  In 
other  words,  training  "creates  the 
potential  to  perform"  (Rouse  and 
Johnson,  1989).  Job  Aiding, 
collectively,  refers  to  those  devices 
with  the  capacity  to  store  and 
retrieve  the  "How",  "What",  and 
"When"  information  pertinent  to  the 
performance  of  a  particular  task. 
Job  aiding,  therefore,  directly 
augments  performance  (Rouse  and 
Johnson,  1989). 

Selecting  from  among  the  wide 
variety  of  training  and  aiding 
alternatives  (and  their  possible 
combinations)  is  difficult  when  a 
myriad  of  interdependent  factors  such 
as  performance-related  effectiveness, 
development/implementation  costs  and 
system  design  impact  must  be 
simultaneously  resolved.  For 
example,  as  an  information  storage 
device,  a  job  aid  facilitates 
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performance  by  reducing  task-related 
memory  requirements.  This,  in  turn, 
reduces  the  training  requirements  for 
that  job  and  generates  the  potential 
for  reducing  immediate  resource 
expenditures.  Training,  on  the  other 
hand,  imparts  more  general  knowledge 
applicable  to  a  variety  of  related 
tasks.  In  this  case,  the  increased 
capital  costs  of  training  a  small, 
multi-disciplinary  work  force  may,  in 
the  long-term,  be  more  cost  effective 
than  the  life-cycle  costs  of 
supporting  a  larger,  more  specialized 
team. 

The  formulation  and  evaluation 
of  these  aiding/training  tradeoffs  is 
a  necessary  component  of  the 
decisions  made  by  MPT  analysts, 
system  designers,  and  personnel 
supervisors  throughout  the  Air  Force. 
To  the  extent  that  these  tradeoffs 
have  been  addressed  in  the  past,  the 
analyses  have  required  many  person- 
years  of  effort.  Often,  the  result 
has  been  a  time-consuming  and 
expensive  effort  that  provided 
insights  which  were  too  late  to  be 
implemented  in  any  substantial  way 
(Rouse  and  Johnson,  1989).  Whether 
for  evaluating  current  AFS  job 
performance,  selecting  among  new 
system  design  alternatives, 
or  ensuring  that  flight-line 
personnel  are  task  qualified,  a 
methodology  for  efficiently  producing 
consistent,  timely,  and  supportable 
aiding/training  decisions  is  a  must. 

In  response  to  this  need,  the  Air 
Force  Human  Resources  Laboratory  at 
Brooks  AFB  has  sponsored  the  Job 
Aiding/Training  Allocation 
Technologies  (JATAT)  program.  The 


purpose  of  JATAT  was  to  develop  a 
conceptual  decision  aiding 
methodology  to  assist  in  identifying 
applicable  tr a i n i ng/ a i d i ng 
alternatives  and  evaluating 
combinations.  The  expected  benefits 
of  such  a  methodology  include  faster, 
more  accurate,  and  auditable 
performance-based  aiding/training 
recommendations. 

OVERALL  DESCRIPTION 

In  a  prior  report.  Rouse  and 
Johnson  (1989)  suggested  three 
computational  approaches  for 
supporting  trade-off  decisions 
between  training  and  job  aiding.  The 
first  approach  involves  compiling 
general  guidelines  for 
training/aiding  decisions  based  on 
cumulative  experience  and 
experiments.  This  results  in  a 
"rule-based"  approach  in  which 
tradeoffs  are  embodied  in  rules  based 
on  mappings  from  task  performance 
requirements  to  training/aiding 
deci sions. 

The  second  approach  involves 
predicting  human-machine  system 
performance  based  on  attributes  of 
specified  training  and  aiding 
alternatives.  This  approach  enables 
the  analyst  to  specify  the 
appropriate  performance  measures  and 
acceptable  levels  of  performance  in 
different  situations  and  with 
different  priorities.  This  requires 
computational  models  that  predict  the 
measures  of  interest  based  on  the 
available  attributes  of  the  training 
or  aiding  alternatives.  These  models 
are  a  more  "visible"  form  of  the 
first  approach  in  that  the  first 
approach  may  "hide"  the  measures  and 
requisite  levels  of  performance  in 
the  rules  or  guidelines. 

If  the  rules,  guidelines,  or 
computational  models  do  not  exist  or 
are  not  adequate,  then  the  analyst 
can  use  simulation  techniques  to 
estimate  performance  measures  of  the 
human-machine  system  using  different 
training  or  aiding  alternatives. 


This  third  approach  is  actually  a 
special  case  of  the  second.  In  this 
case,  the  analyst  must  develop  or 
tailor  a  simulation  model  to  predict 
the  chosen  measures  of  performance. 

The  methodology  described  in 
this  paper  encompasses  these  three 
approaches,  placing  them  in  the 
larger  context  of  a  training/aiding 
trade-off  analysis  (see  Figure  1). 
Steps  1  and  2  of  the  method  reflect  a 
typical  systems  engineering  approach 
to  the  analysis.  Step  3  indicates  a 
human-centered  approach  to 
identifying  the  requirements  that 
will  be  addressed  in  the  analysis. 
Steps  4  through  15  of  the  method  (in 
which  the  analyst  determines  the 
alternatives  and  formulates, 
analyzes,  and  integrates  the 
tradeoffs)  describe  an  ordered 
approach  to  the  complex  problem  of 
analyzing  multiple,  interdependent 
tradeoffs  between  aiding  and  training 
the  human. 

Although  presented  as  an  ordered 
list  in  Figure  1,  these  steps  are  not 
necessarily  sequential.  Some  of  the 
steps  may  be  repeated  several  times 
as  the  analyst/designer  works  through 
the  tradeoffs  under  various 
conditions  and  with  various 
combinations.  The  following  sections 
discuss  these  steps  in  detail. 

INDIVIDUAL  STEP  DESCRIPTION 

IDENTIFY  TASKS 

1.  Understand  the  Job.  In  the 
context  of  analyzing  job 
aiding/training  tradeoffs,  the 
analyst  must  understand  three 
different  aspects  of  the  job;  the 
tasks  involved  in  the  job,  the 
equipment  used  in  the  job,  and  the 
personnel  expected  to  do  the  job. 
This  knowledge  is  necessary  to 
determine  the  job  requirements  and 
system  constraints  that  must  be 
satisfied.  Obviously,  complete 
knowledge  of  these  variables  for  all 
tasks  is  unrealistic.  In  fact,  the 
required  level  of  understanding  is 
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directly  dependent  upon  the  problem 
at  hand.  For  example,  a  decision  to 
train  or  aid  personnel  to  perform  a 
job  requires  far  less  detailed 
information  than  a  decision  among 
combinations  of  a  specific  task. 

2.  Decompose  via  Task  Taxonomy. 
The  second  step  is  to  further 
decompose  the  tasks  into  more 
primitive  tasks,  referred  to  as 
subtasks  or  activities.  This 
decomposition  defines  the  level  of 
granularity  for  subsequent  steps  (the 
assessment  of  the  human's 
limitations,  abilities,  and 
preferences).  A  task  taxonomy  is 
useful  in  this  step,  particularly  if 
the  human's  limitations,  abilities, 
and  preferences  are  readily 
determined  for  the  task  elements  in 
the  taxonomy. 

ASSESS  LIMITATIONS.  ABILITIES.  AND 
PREFERENCES 

3.  Assess  Human  Limitations. 
Abilities,  and  Preferences.  In  this 
step,  the  analyst  determines  those 
characteristics  of  the  human  in  the 
system  that  either  require  (through 
human  limitations)  or  influence 
(through  human  abilities  and 
preferences)  training/aiding 
decisions.  It  is  this  focus  on  the 
human  capabilities,  limitations,  and 
preferences  in  the  system  that  makes 
this  a  human-centered  approach. 

This  assessment  draws  its 
primary  input  from  the  task 
decomposition  in  the  previous  step, 
which  provides  an  "index"  for  human 
limitations,  capabilities,  and 
preferences.  In  subsequent  steps, 
these  assessments  will  be  used  to 
identify  training  and  aiding 
alternatives.  A  task  decomposition 
that  is  too  coarse  leads  to 
identifying  general  human  limitations 
that  are  not  sensitive  to  the 
aiding/training  alternatives 
available.  A  task  decomposition  that 
is  too  fine  grained  leads  to 
identifying  human  limitations  that 


require  premature  detailed  design  of 
aiding/training  alternatives  to 
evaluate. 

DETERMINE  ALTERNATIVES 

4.  Map  Limitations.  Abilities, 
and  Preferences  to  a  Taxonomy  of 
Training  Alternatives. 

5.  Map  Limitations.  Abilities, 
and  Preferences  to  a  Taxonomy  of 
Aiding  Alternatives.  Through  Steps  4 
and  5,  the  analyst  uses  the 
limitations  and  abilities  identified 
in  Step  3  to  guide  the  identification 
and  selection  of  alternative  training 
and  aiding  techniques.  This  is  done 
by  identifying  the  knowledge  and 
skill  requirements  of  a  task  and 
mapping  the  required  changes  in 
knowledge  and  skills  to  candidate 
aiding/training  methods  through 
guidelines.  The  mapping  is  guided  by 
available  expert  heuristics  or 
empirically  developed  guidelines. 

From  a  pragmatic  perspective, 
other  factors  may  also  go  into  this 
process,  such  as  resource 
availability  and  existing  training  or 
aiding  techniques  for  this  or  similar 
jobs.  Depending  on  the  maturity  of 
the  analysis  and  the  expertise  of  the 
analyst,  these  considerations  may 
either  prematurely  constrain  the 
solution  space  (early  in  the 
analysis)  or  provide  timely  guidance 
leading  to  practical  solutions  (later 
in  the  analysis) . 

FORMULATE  TRADEOFFS 

6.  Make  Obvious  Choices.  In 
this  step,  the  analyst  selects  among 
training/aiding  alternatives  that  are 
straight-forward  and  require  no 
additional  analysis.  This  step 
allows  for  the  situation  in  which 
part  of  the  problem  is  easily 
addressed  by  conventional  solutions. 
For  example,  printed  procedural  job 
aids  may  be  an  obvious  solution  for  a 
task  which  is  similar  to  one  already 
using  that  type  of  aid  extensively. 
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Clearly,  these  choices  depend 
upon  the  expertise  of  the  analyst  as 
well  as  the  data  and  tools  available 
to  the  analyst.  A  relatively  novice 
analyst  may  be  unable  to 
independently  make  obvious  choices, 
but  may  be  able  to  rely  upon  tools 
such  as  heuristic  guidelines, 
decision  flow  charts  (Booher,  1978), 
or  expert  judgement  models  (Irvin, 
Blunt,  &  Lamb,  1988)  for  making  broad 
categorical  decisions  (e.g.,  train, 
aid,  both,  or  either).  A  more 
experienced  analyst  may  also  want  to 
use  these  tools  to  verify  their 
choices. 

Making  the  obvious 
training/aiding  choices  now,  however, 
does  not  remove  them  from  further 
consideration.  Their 
interdependencies  must  still  be 
considered  in  later  steps. 

7.  Coalesce  Interdependent 
Tradeoffs.  To  this  point  in  the 
analysis,  the  number  of  viable 
alternatives  has  been  relatively 
unlimited.  However,  once  obvious 
choices  have  been  made,  subsequent 
analyses  can  be  quite  extensive. 
Therefore,  it  is  usually  necessary  to 
narrow  down  the  number  of  candidate 
solutions  by  grouping  training  and 
aiding  alternatives  according  to 
their  interdependent  relationships 
and  characteristics. 

For  example,  a  particular  task 
element  may  suffer  from  a  limitation 
that  may  be  addressed  by  one  of  three 
alternatives:  training  alone,  aiding 
alone,  or  some  combination  of 
training  and  aiding.  It  is  likely 
that  these  task  elements  will  be 
functionally  or  temporally 
interrelated.  Similarly  the  training 
and  aiding  alternatives  will  probably 
have  interdependent  interdependencies 
and  coalesce  the  training/aiding 
alternatives  into  a  smaller  set  for 
subsequent  analysis. 

ANALYZE  TRADEOFFS 

8.  Choose  Measures  o?^'^ 


Performance.  Accurately  evaluating 
the  resultant  training/aiding 
alternatives  requires  selecting  the 
appropriate  performance  measures. 
These  measures  are  clearly  domain 
dependent.  Cost,  for  example,  is  a 
basic  measure  of  performance  common 
to  all  domains;  although  its 
importance  will  vary  accordingly. 
Other  examples  include  time  to 
perform,  probability  or  number  of 
errors,  mean  time  between  failures, 
etc. 

The  choice  of  performance 
measures  is  also  influenced  by  the 
available  modeling  tools  and  modeling 
expertise  of  the  analyst.  While  a 
more  experienced  analyst  may  choose 
to  tailor  the  available  modeling 
tools  or  develop  new  models  to 
produce  a  variety  of  performance 
measures,  a  novice  will  probably  have 
to  choose  among  "pre-determined" 
models  that  are  readily  available, 

9.  Choose  Input/Output 
Representations.  To  compare 
training/aiding  alternatives,  an 
input/output  (I/O)  representation 
(i.e.,  a  model)  must  be  chosen  that 
can  produce  the  selected  measures  of 
performance.  The  I/O  representation 
must  reflect  realistic  inputs  from 
available  data  and  the  desired 
outputs  including  the  performance 
measures. 

Once  again,  the  experience  level 
of  the  analyst  strongly  influences 
the  extent  of  this  step.  While  a 
more  experienced  analyst  may  be  able 
to  adapt  existing  models  or  develop 
new  ones,  the  choice  of  I/O 
representation  for  novice  will  more 
likely  follow  directly  from  the 
choice  of  performance  measure. 

10.  Identify  Requisite 
Structures  and  Parameters  for 
Representations.  Employing  the 
chosen  I/O  representation  frequently 
requires  modeling  the  human  as  an 
integral  component  of  the  system.  In 
doing  so,  it  may  be  necessary  to 
determine  the  structures  and 


parameters  that  represent  how  the 
human  performs  the  task.  If  the 
analysis  includes  only  aiding 
alternatives,  these  requirements  may 
be  essentially  constant  throughout 
the  analysis.  In  analyses  that 
include  training  alternatives,  these 
requirements  will  vary  to  simulate 
the  impact  of  different  training 
alternatives. 

11.  If  Necessary.  Represent  the 
Learning  Process.  In  some  analyses, 
the  performance  measures  may  be 
sensitive  to  the  human  process  of 
acquiring  knowledge  and  skills.  In 
these  cases,  the  learning  process 
must  be  reflected  in  the  model.  This 
representation  may  be  as  simple  as 
retrieving  data  from  a  database  or  as 
complex  as  employing  learning  curves 
or  learning  process  models. 

12.  Apply  Methods  of  Analysis  to 
Representations.  This  step  invokes 
the  targeted  analysis;  input  data  is 
supplied,  the  model  is  exercised,  and 
performance  data  is  collected  for 
each  of  the  training/aiding 
alternatives  of  interest. 

13.  Interpret  Results.  Next, 
data  collected  during  the  previous 
step  is  analyzed  and  interpreted  in 
the  context  of  selected  analyses. 
This  step  may  be  repeated  several 
times  in  conjunction  with  steps  10 
through  12  as  the  analyst 
investigates  the  effects  of  various 
assumptions  or  the  sensitivity  of  the 
performance  measures  to  variations  of 
the  parameters  within  the  model. 

INTEGRATE  TRADEOFFS 

14.  Compile  Assumptions  and 
Consequences  of  Tradeoffs.  In  an 
extensive  analysis  with  a  number  of 
different  tasks  and  training/aiding 
alternatives,  organizing  the 
assumptions  and  consequences  of  the 
trade-off  analyses  is  a  large 
bookkeeping  task.  The  purpose  of 
this  step  is  to  compile  all  th?35 


conmion  aiding/training  alternative 
characteristics  and  decisions  in 
order  to  implement  the  predetermined 
aggregation  guidelines  in  the 
following  step. 

15.  Form  Sets  of  Tradeoffs  with 
Consistent  Assumptions  and 
Consequences.  In  the  final  step  of 
the  methodology,  the  training/aiding 
alternatives  are  integrated  into  sets 
satisfying  the  requirements  developed 
from  the  human  limitations, 
abilities,  and  preferences  identified 
in  Step  3.  In  addition  to  satisfying 
these  requirements,  each  set 
incorporates  the  common  assumptions 
and  consequences  (i.e.,  learning  and 
retention  abilities  of  humans  or 
productivity  improvements  with  job 
aiding)  identified  in  the  previous 
step. 

While  most  analysts  will 
probably  not  have  the  final  decision¬ 
making  authority  necessary  to 
implement  the  recommendations 
produced  in  a  JATAT  analysis,  this 
methodology  generates  a  logical 
justification  supporting  these 
recommendations.  The  purpose  of  this 
step,  therefore,  is  to  compile  a 
clear,  coherent  summary  of  that 
justification. 

SUPPORTING  DATA  TYPES 

Similar  to  other  decision  aiding 
paradigms,  the  JATAT  methodology  is 
extremely  data  dependent.  The 
type/ format  of  data  required  by  each 
component  of  the  decision  process 
varies  as  a  function  of  the 
characteristics  of  that  intermediate 
decision  construct.  For  JATAT,  these 
data  requirements  can  be  classified 
as  archival  data,  user-provided 
contextual  information,  and  expert 
rules/heuristics. 

Archival  data  is  the  factual 
personnel,  task,  and  equipment 
information  typically  collected 
through  occupational  surveys  or 
compiled  from  system  operation  and 
maintenance  documents.  This  includes 


such  data  as  task/sub-task  listings, 
performance  times/probabilities/and 
requirements,  task  and  equipment 
complexity,  aptitude/bxperience,  and 
training  indicators.  In  the  JATAT 
aiding/training  decision  support 
paradigm,  archival  data  serve  as  both 
the  target  data  to  be  processed  by 
the  JATAT  models,  and  as  the  user's 
source  of  domain  knowledge. 
Currently,  this  information  resides 
in  a  miscellany  of  data  sources  which 
include  the  Occupational  Research 
Data  Bank  (ORDB),  AF  Regulations, 
career  development  training  manuals. 

Air  Training  Command  Plans  of 
Instruction  (POI's),  and  equipment 
technical  manuals. 

Contextual  information  is  the 
abstract,  highly  situation  specific 
data  which  represents  the  background 
scenario  of  the  archival  data.  This 
information  is  primarily 
characterized  by  its  relational 
nature.  It  encompasses  those  factors 
which  preceded  (and  subsequently, 
effected)  the  aiding/training 

analysis,  and  the  environmental 
ramifications  (i.e.,  political, 
temporal,  and  resource-related)  of 
the  candidate  aiding/training 

solutions.  Due  to  the  context 
specific/temporal  nature  of  this  data 
the  JATAT  aiding  /training  decision 
analyst  is  the  sole  source  of  this 
information. 

The  aiding/training  expert 
rules/heuristics  contain  the 
knowledge  which  guides  the  analysis 
process,  generates  aiding/training 
recommendations,  and  simulates  target 
system  performance.  This  data  is 
primarily  in  the  form  of  taxonomies 
and  expert  rules  resident  in  the 
system  knowledge  bases.  Included  in 
this  category  are  potential 
training/aiding  methods,  training 
philosophies,  operational/system 
knowledge  requirements,  modeling 
packages,  input/output  parameters, 
etc.  This  knowledge  must  be 
established  during  the  system  design 
and  development  phases  and  is 
obtained  through  knowledge36 


acquisition  sessions  with  relevant 
domain  experts. 

LESSONS  LEARNED 

In  order  to  verify  the 
robustness  of  the  newly  developed 
JATAT  methodology,  we  preformed  two 
aiding/training  trade-off  analyses  of 
pre-selected  Air  Force  Specialty 
tasks  based  on  available  operational 
data.  The  lessons  learned  from  this 
experience  were  two-fold.  First,  the 
JATAT  methodology  developed  during 
this  effort  successfully  serves  as  a 
generic  decision  framework  for 
supporting  accurate,  auditable 
aiding/training  trade-off  analyses. 
And  second,  current  Air  Force 
occupational  survey  procedures  must 
be  reevaluated  in  order  to  support 
integrated  decision  support  models, 
such  as  JATAT.  This  reevaluation 
process  must,  at  a  minimum,  address 
the  issues  of  data  availability  and 
standard  task  definitions. 

DATA  AVAILABILITY 

Data  availability,  in  this  case, 
is  defined  in  terms  of  the  data's 
format  (i.e.,  how  closely  the 
existing  data  format  conforms  to  that 
required  by  the  decision  models 
defines  the  degree  of  pre-processing 
necessary)  and  the  medium  within 
which  the  data  resides  (i.e., 
hardbound  copies  vs.  a  computer-based 
data  bank).  Each  level  of 
extrapolation,  interpolation  and 
media  transformation  required  to 
apply  the  data  to  the  methodological 
framework,  in  essence,  decreases  the 
inherent  utility  of  the  data. 
Unfortunately,  current 
data  sources  vary  widely  along  both 
of  these  dimensions.  Therefore, 
commitment  and  effort  will  be 
required  to  structure  available  data 
sources  to  more  directly  support  the 
integrated  environment  of  the  many 
new  decision  aiding  models. 

STANDARD  TASK  DEFINITIONS 


The  importance  of  formulating 
universally  accepted,  standardized 
task  definitions  (i.e.,  terminology 
and  level  of  resolution)  represents  a 
single,  cogent  conclusion  drawn  from 
three  independent  issues.  First,  the 
accuracy  of  a  decision  aiding 
application  is  directly  dependent 
upon  the  mapping  of  an  unfamiliar 
task  to  a  well -understood  taxonomy. 
Complicating  this  mapping  by 
representing  tasks  at  various  levels 
of  abstraction  decreases  the  power 
and  success  of  the  aiding/training 
recommendations.  The  second  issue 
supporting  the  pursuit  of  a 
standardized  task  definition  effort 
is  that  current  task  listings  lack 
consistency  in  their  granularity  of 
task  specification.  For  example,  the 
Air  Force  Occupational  Research 
DataBank  specifies  both  form 
completion  and  propulsion  system 
trouble-shooting  at  the  task  level. 

No  explicit  standard  is  established 
regarding  time  required,  number 
and/or  type  of  activities  involved, 
focus  of  aiding/training  solutions, 
etc.  This  inconsistency  among 
critical  task  dimensions  also 
complicates  the  process  of 
transitioning  information  among 
different  decision  models.  Recent 
Air  Force  emphasis  on  MPT  model 
integration,  the  third  issue,  is 
directed  toward  developing 
consistency  and  relatedness  among  the 
numerous  decision  models.  Definition 
standardization  within,  and  across, 
models  and  databases  is  essential  to 
facilitate  such  a  concept. 

SUMMARY 

This  paper  describes  a  generic 
methodological  framework  for  making 
job-aiding  and  training  trade-off 
decisions  about  a  wide  variety  of 
operational  Air  Force  tasks.  These 
tasks  may  be  anything  from  trouble¬ 
shooting  complex  aircraft  avionics  to 
patrolling  airbase  perimeters, 
currently,  these  analyses 
results,  inherent  to  the  JATA+37 


decision  process,  are  purposely 
grounded  in  human -system  performance 
attributes  and  predictions. 
Subsequent  efforts  will  be  needed  to 
address  the  issues  of  cost  and 
operational  constraints. 

The  potential  impact  of  a 
computer-based  application  of  the 
JATAT  decision  aiding  methodology  on 
Air  Force  Manpower,  Personnel,  and 
Training  (MPT)  issues  is  the  ability 
to  provide  logical  arguments  for  the 
construction  of  job-aids  and 
empirical  justification  for  the 
manner  in  which  we  decide  to  train 
specified  tasks.  While  the  successes 
of  these  efforts  make  no  implications 
regarding  the  maturity  and  readiness 
of  the  technology  necessary  for 
implementing  the  JATAT  methodology, 
they  do  indicate  that  the  solution  is 
tenable.  It  will  be  further  research 
into  the  areas  of  the  input  factor 
relationships  within  the 
aiding/training  domain,  continued 
trade-off  decision  methodology 
formulation,  supporting  database 
development,  predictive  model 
generation,  and  the  design  of 
decision  support  system  functionality 
that  will  help  bring  the  JATAT 
concept  to  reality. 
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IDENTIFY  TASKS 


1.  UNDERSTAND  THE  JOB 

2.  DECOMPOSE  VIA  TASK  TAXONOMY 


ASSESS  HUMAN  LIMITATIONS,  ABILITIES 
AND  PREFERENCES 

3.  ASSESS  HUMAN  LIMITATIONS.  ABILITIES 
AND  PREFERENCES 


DETERMINE  ALTERNATIVES 

4.  MAP  LIMITATIONS.  ABILITIES,  AND  PREFERENCES  TO 

A  TAXONOMY  OF  TRAINING  ALTERNATIVES 

5.  MAP  LIMITATIONS,  ABILITIES.  AND  PREFERENCES  TO 

A  TAXONOMY  OF  AIDING  ALTERNATIVES 


FORMULATE  TRADE-OFFS 

6.  MAKE  OBVIOUS  CHOICES 

7.  COALESCE  INTERDEPENDENT  TRADE-OFFS 


ANALYZE  TRADE-OFFS 

CHOOSE  MEASURES  OF  PERFORMANCE 
CHOSE  INPUT/OUTPUT  REPRESENTATIONS 
IDENTIFY  REQUISITE  STRUCTURE  AND  PARAMETERS 
FOR  REPRESENTATIONS 

IF  NECESSARY,  REPRESENT  THE  LEARNING  PROCESS 
APPLY  METHODS  OF  ANALYSIS  TO  REPRESENTATIONS 
INTERPRET  RESULTS 


INTEGRATE  TRADE-OFFS 


COMPILE  ASSUMPTIONS  AND  CONSEQUENCES  OF 
TRADE-OFFS 

FORM  SETS  OF  TRADE-OFFS  WITH  CONSISTENT 
ASSUMPTIONS  AND  CONSEQUENCES 


239 


EXPANDING  THE  USEFULNESS 
OF 

ANALYSIS  AND  DESIGN  INFORMATION 
Jeff  Clark 


ABSTRACT 

iSD  methodology  used  on  large,  complex  training  programs  generates  enormous 
amounts  of  information.  All  the  information  must  be  organized  and  put  in  to 
formats  that  are  usable  by  managers,  subject  matter  experts  and  instructional 
designers.  The  purpose  of  creating  this  information  is  to  facihtate  decision 
making,  and  to  find  and  document  the  most  effiaent  and  effective  training 
structure  attainable  given  the  assets  ai\d  constraints.  Now,  using  newer  computer 
technology  and  database  software,  analysis  and  design  information  can  also  be 
used  to  create  a  wide  range  of  training  products.  Thest  products  include  job 
survey  forms,  self  evaluation  checklists,  job  aids,  academic  tests,  and  job 
performance  measures.  This  dramatically  increases  the  value  of  analysis  and 
design  information. 

JEFF  CLARK  is  an  Instructional  Psychologist  at  FOCUS  Learning  Corporation. 
He  received  his  Ph.D.  in  Educational  Psychology  from  The  University  of  Utah. 
For  the  past  six  years  he  has  worked  with  training  organizations  at  nuclear  power 
plants  to  design  and  develop  technical  training  programs.  Prior  to  his  work  in  the 
nuclear  industry.  Dr.  Clark  served  as  a  training  consultant  to  organizations  in 
business  and  the  military. 
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EXPANDING  THE  USEFULNESS 
OF 

ANALYSIS  AND  DESIGN  INFORMATION 


Introduction 

Traditionally,  analysis  and  design 
data  have  been  used  to  describe  jobs, 
identify  the  skills,  abilities,  and 
knowledge  required  for  competent 
performance  of  the  job,  and  to  define 
the  structure  of  training  programs. 
Rigorous  use  of  these  processes  has 
generally  resulted  in  more  effective 
and  efficient  training. 

Now,  we  are  able  to  go  beyond  those 
classical  uses  of  analysis  and  design 
data  in  several  important  ways. 
First,  we  can  make  the  training  mere 
consistent  with  actual  job  performance 
by  using  software  to  track  the  link 
between  the  job/task  analysis, 
objectives,  and  the  finished  training 
program.  This  link  lets  the  developer 
see  what  job/task  analysis  information 
has  not  been  used  to  develop 
objectives,  and  conversely  what 
objectives  have  been  created  that  do 
not  have  a  link  to  the  job/task 
analysis.  This  enables  the  designer 
to  control  the  content  of  the  training 
program  and  provides  a  rational  basis 
for  deciding  what  shoxild  and  should 
not  end  up  in  the  training  program. 

In  addition,  the  designer  can  track 
where  in  the  training  program  specific 
job/task  analysis  components  are 
taught.  If  equipment  is  changed,  the 
designer  can  quickly  assess  the 
impact  on  existing  training. 

Another  important  new  capability  is 
the  creation  of  useful  training 
products  directly  from  an  analysis  and 
design  database.  Typically,  analysis 


and  design  data  have  ’)iien  used  and 
then  forgotten.  But  computer 
technology  with  larger  storage  devices 
and  faster  processing  speeds,  is  giving 
us  new  opportunities  to  make  use  of 
this  valuable  information.  Some  of 
the  products  that  can  be  generated 
directly  from  analysis  and  design  data 
include  job  survey  forms,  task 
analysis  worksheets,  job  aids,  self 
evaluation  checklists,  academic  tests, 
and  job  performance  measures. 

These  documents  are  created 
independently  of  one  anotiier  rnd  yet 
they  all  share  common  information. 
For  example  a  task  statement  can 
show  up  in  a  job  analysis,  an 
objective,  a  job  aid,  and  on  a  job 
performance  measure.  If  the  task 
changes,  then  these  documents  are 
probably  effected.  Gradually,  these 
documents  get  out  of  phase  with  one 
another  «nd  inconsistencies  creep  into 
the  information  with  the  resulting 
inconsistencies  in  the  training. 

There  are  several  ways  of  dealing 
with  these  inconsistencies.  One  is  to 
ignore  them  imtil  they  become  an 
unavoidable  problem.  Usually  this 
strategy  is  used  by  those  that  view 
analysis  and  design  information  as  a 
means  to  an  end  and  have  thrown 
the  analysis  and  design  data  away  cr 
put  it  in  a  file  somewhere.  When  the 
training  gets  out  of  phase  with  the 
job  it  purports  to  train,  analysis  and 
design  activities  are  started  again. 

Another  solution  has  been  to  institute 
a  controlled  documents  policy. 
Procedures  are  put  into  place  that 
specific  how  and  when  changes  are 
made  to  important  training 
documents.  However,  these 

doemnents  are  created  separately 
using  a  combination  of  database  and 
word  processing  software.  Updates  to 
one  document  doesn’t  change  the 
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content  of  related  documents.  So 
document  maintenance  becomes  a 
very  labor  intensive  process  and  still 
doesn’t  completely  solve  the  problem. 

A  third  and  newer  possibility  is  to 
have  all  the  information  required  to 
produce  these  documents  in  a  set  of 
integrated  software  programs. 
(Integration  in  this  context  means 
that  job/task  analysis,  objectives, 
program  design,  and  associated 
information  all  reside  in  one 
database.)  It  is  now  possible  to  do 
this,  and  several  important  things 
happen  as  a  result.  The  analysis  and 
design  database  can  be  used  to 
facilitate  program  revisions  and 
maintenance.  It  can  also  be  used  to 
generate  training  documents.  When 
updates  are  made  they  are  reflected 
in  many  different  parts  of  the 
program  at  the  same  time  and  the 
training  products  are  kept  current. 


Specialized  Software 

To  maintain  the  documents  and  get 
them  from  one  database,  two  critical 
things  have  to  occur.  There  must  be 
a  software  package  that  can  handle 
the  complex  relationships  that  exist 
between  the  different  types  of  training 
information  and  there  must  be  a  plan 
for  the  structure  and  organization  of 
the  database.  At  a  minimum  the 
software  should  have  these 
capabilities: 

1.  Represent  the  job  and  task 
analysis  in  a  flexible 
hierarchical  format  that  defines 
relationships  between  the  job, 
its  duty  areas,  tasks, 
performance  steps,  skills,  and 
knowledge  statements. 


2.  Maintain  the  link  between  the 
job/task  analysis,  objectives,  and 
program  design, 

3.  Attach  associated  information, 
like  procedures,  references,  and 
equipment  to  tasks,  objectives, 
and  program  design, 

4.  Flexible  report  generation 
format  that  allows  the  creation 
of  custom  reports  and  the 
revision  of  standard  reports, 

5.  Handle  both  relational  and 
hierarchical  data, 

6.  Facilitate  the  effective  and 
efficient  application  of  the  ISD 
process, 

7.  Provide  security  with  password 
access  to  the  database. 

Currently,  there  are  few  off  the  shelf 
software  packages  that  meet  these 
requirements,  but  as  the  power  of  the 
integrated  database  concept  takes 
hold  in  training  organizations,  there 
will  be  more  and  more  to  choose 
from. 


Organizing  the  Database 

Assuming  the  software  is  available, 
the  structure  of  the  database  must  be 
planned  and  organized.  This  means 
determining  how  the  information 
should  be  structured  so  that  useful 
training  products  can  be  generated 
directly  from  the  database.  This 
usually  starts  by  making  decisions 
about  the  basic  components  of  the 
database.  (For  purposes  of  discussion, 
the  basic  components  can  be  thought 
of  as  the  job/task  analysis,  the 
objectives,  and  the  program  design.) 
Typical  decisions  about  these 
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components  include;  how  the  job  and 
task  analysis  will  be  organized,  how 
the  objectives  will  be  organized, 
should  conditions  and  standards  be 
written  with  the  objectives  or  should 
they  come  from  a  table,  and  how  do 
objectives  get  sequenced  into  lessons. 
It  is  important  to  think  about  these 
issues  at  the  start  of  a  project  so  an 
efficient  process  can  be  set  up  to 
collect  the  appropriate  information. 

Then,  decisions  are  made  about  the 
information  that  will  be  associated 
with  the  basic  components.  The  types 
of  associated  information  is  usually 
derived  from  an  examination  of  the 
existing  training  materials.  For 
example,  information  on  a  job 
performance  measure  usually  contains 
such  items  as  a  task  statement, 
performance  steps,  conditions, 
standards,  tools,  equipment,  and 
procedures,  as  well  as  a  variety  of 
standard  text.  This  information  is 
identified  and  decisions  are  made  on 
how  to  organize  the  database  to 
produce  the  job  performance  measure. 

Once  the  database  has  been  planned 
and  important  training  products 
identified,  the  data  is  collected  and 
entered  into  the  computer.  If  the 
analysis  and  design  data  is  being 
generated  from  scratch  then  different 
steps  are  taken  than  if  it  is  generated 
from  existing  materials.  Generally, 
when  existing  materials  are  used 
there  are  many  inconsistencies  to 
resolve. 

In  general,  the  process  of  organizing 
a  database  has  the  following  steps: 

1.  Identify  the  desired  outputs, 

2.  Identify  the  information  needed 
on  the  outputs  and  specify  an 
acceptable  format. 


3.  Relate  the  information  to  the 

appropriate  step  in  the  ISD 
process, 

4.  Create  a  process  to  collect 

information, 

5.  Enter  the  data, 

6.  Create  the  reports. 

This  process  is  something  new  for 

most  instructional  designers.  It 
requires  a  careful  analysis  of  the 
information  requirements  of  the 
training  program  and  an  ability  to 
translate  those  requirements  into  a 
database  structure  that  reflects  the 
ISD  process.  This  process  is  different 
from  the  traditional  activities  of  an 
instructional  designer,  but  is 
something  instructional  designers  will 
become  more  familiar  with  as 
instructional  databases  become  a 
reality  in  large  training  organizations. 

Implementation  Considerations 

In  order  to  make  this  approach 
successful  there  are  several  important 
management  and  organizational 
challenges  that  must  be  overcome. 
There  has  to  be  strong  backing  by 
management  and  a  commitment  to 
the  integrated  database  concept.  This 
leadership  will  establish  the  first 
project  using  this  methodology  as  a 
high  priority  and  secure  and  deliver 
vital  resources  to  the  project  when 
necessary. 

Several  jobs  in  the  organization  will 
change  significantly  if  this  approach 
is  implemented.  For  example, 
someone  has  to  become  responsible 
for  the  database.  They  will  have  to 
help  set  policies  and  guidelines  about 
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the  structure  and  content  of  the 
database.  At  a  minimum  this 
requires  extensive  knowledge  of  the 
software,  the  ISD  process,  and  the 
ability  to  interact  with  instructors, 
subject  matter  experts  and 
instructional  designers. 


2.  Having  software  that  will 
handle  the  unique  training 
information  requirements, 

3.  Obtaining  management 
commitment,  a  solid 
administrative  plan,  and  new 
types  of  resources. 


The  Future 

If  we  are  able  to  generate  training 
products  directly  from  one  integrated 
database,  then  there  are  other 
important  functions  that  could  be 
added.  For  example  it  is  possible  to 
link  a  test  generator  to  this  database 
and  use  the  objectives  to  help  us 
write  a  bank  of  test  questions. 
Furthermore  we  can  use  the  structure 
of  the  training  program  to  generate 
tests  from  the  test  bank.  It  also 
follows  that  you  can  track  a  student 
through  the  training  program  or 
manage  student  records  based  on  the 
program  design.  In  addition,  we 
should  be  able  to  let  a  student  into 
the  database  to  down  load  lessons,  job 
aids  or  JPMs  right  at  their 
workstation. 


Summary 

We  now  have  the  technology  to  make 
analysis  and  design  information  much 
more  useful  than  it  has  been  in  the 
past.  But  to  realize  these 
possibilities,  we  have  to  change  the 
way  we  think  about  training 
information  .  The  keys  to  making 
analysis  and  design  information  more 
usefal  are: 

1.  Knowing  how  to  plan  the 
structure  and  organization  of  a 
training  database, 
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CURRENT  TRENDS  IN  SYSTEMS  DEVELOPMENT: 
THE  EMERGENCE  OF  CASE  TECHNOLOGY 


Herman  P.  Hoplin 
Abstract 

With  the  current  trend  toward  the  use  of  Computer-Aided  Software 
Engineering  (CASE)  in  Systems  Development  and  its  related  aspects  of 
prototyping,  end-user  development,  and  code  generators,  the  paper  focuses  on 
these  rapidly  emerging  developments.  CASE  has  become  more  and  more  vital 
to  both  large  and  small  projects  in  both  the  Military  and  Civilian  Communities. 
Industry  encourages  colleges  and  universities  to  develop  software  engineering 
programs  based  upon  automated  tools  in  Instructional  Systems  Development. 
Particular  focus  is  on  increasing  the  productivity  of  current  and  future  software 
engineers.  CASE  technology  is  seen  as  one  of  the  ways  in  which  this  can  be 
accomplished.  Now  that  the  management  of  computerized  information  systems 
has  become  an  integral  function  within  the  business  establishment,  it  also  is 
undergoing  the  same  far-reaching  transformations  caused  by  computer 
automation.  CASE  tools  are  being  designed  with  the  business  analyst  in  mind, 
not  the  programmer.  An  ideal  set  of  CASE  tools  would  not  only  help  the  user 
analyze  system  requirements  and  design  the  program  modules  to  satisfy  those 
requirements,  it  would  also  automatically  generate  the  code,  perform  version- 
control  and  maintenance  functions,  and  allow  the  user  to  extract  any  drawing  or 
specification  or  code  block  from  the  database  for  inclusion  in  the 
documentation.  For  maximum  versatility,  the  ideal  toolset  would  allow  the  user 
to  import  data  from  or  export  data  to  other  vendors’  CASE  tools  and,  by 
means  of  reverse-engineering  techniques,  help  update  existing  programs  and 
systems  that  were  not  originally  designed  with  CASE.  Unfortunately,  the  ideal 
CASE  toolset  does  not  currently  exist.  The  author  analyzes  CASE  as  it 
currently  stands;  then  looks  at  the  benefits  and  drawbacks  of  CASE  technology. 
Also  addressed  is  what  users  are  looking  for  in  CASE  and  how  they  evaluate 
their  current  CASE  applications.  Finally,  the  author  looks  at  the  future  of 
CASE,  focusing  on  areas  with  high  potential  for  CASE  usage. 
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CURRENT  TRENDS  IN  SYSTEMS 
DEVELOPMENT: 

THE  EMERGENCE  OF 
CASE  TECHNOLOGY 


Herman  P.  Hoplin 

INTRODUCTION 

More  and  more,  Computer-aided 
Software  Engineering  (CASE)  tools 
are  being  designed  with  the  business 
analyst  in  mind,  not  the 
programmer.  An  ideal  set  of  CASE 
tools  would  not  only  help  the  user 
analyze  system  requirements  and 
design  the  program  modules  to 
satisfy  those  requirements,  it  would 
also  automatically  generate  the  code, 
perform  version-control  and 
maintenance  functions,  and  allow  the 
user  to  extract  any  drawing  or 
specification  or  code  block  from  the 
database  for  inclusion  in  the 
documentation.  For  maximum 
versatility,  the  ideal  toolset  would 
allow  the  user  to  import  data  from 
or  export  data  to  other  vendors’ 
CASE  tools  and,  by  means  of 
reverse-engineering  techniques,  help 
update  existing  programs  and 
systems  that  were  not  originally 
designed  with  CASE.  Unfortunately, 
the  ideal  CASE  toolset  does  not 
currently  exist.  Considerable  gaps  in 
the  sequence  of  CASE  design 
remain  and  the  necessary  technology 
or  industry-wide  standard  to  fill 
these  gaps  is  lacking.  CASE  tool 
vendors  are  well  aware  of  these 
gaps  and  are  working  to  develop 
more  sophisticated  tools  to  integrate 
these  more  closely. 

At  present,  the  CASE  toolsets  are 


strongest  in  the  requirements-analysis 
and  the  module-design  phases,  the 
so-called  front-end  tools.  Recently, 
reverse-engineering  tools  for  the 
maintenance  phases  have  started  to 
become  a  reality.  The  biggest  voids 
still  show  up  in  the  code-generation 
phase  and  intertool  communication 
capabilities. 

All  CASE  toolsets  provide  a  means 
for  fully  and  accurately  specifying 
what  a  software  system  must  do. 
The  system  requirements  phase  is 
the  key  to  success  or  failure.  At 
this  stage,  it  is  feasible  and  relatively 
inexpensive  to  change  the  design  to 
satisfy  new  customer  requirements 
and  to  ensure  that  the  logic  of  the 
design  is  correct  and  allows  for 
future  changes.  The  ease  or 
difficulty  with  which  you  can  make 
such  changes  depends  on  the 
complexity  of  the  project. 

In  a  large  project,  it  is  not  easy  for 
any  one  designer  to  grasp  the  whole 
picture  of  what  the  system  must  do. 
CASE  tools  allow  as  many  people  as 
necessary  to  work  on  different  parts 
of  the  design;  built-in  functions 
check  the  completeness  and 
consistency  of  each  part  of  the 
systems  requirements.  Because 
CASE  tools  give  you  instant  access 
to  all  sections  of  a  design,  you  can 
review  the  interfacing  information  to 
ensure  that  your  own  section  joins 
accurately  and  smoothly  to  other 
sections. 

CASE,  which  was  once  the  software 
develop)er’s  impossible  dream,  is  now 
a  reality.  One  practitioner  states 
that  CASE  technologies  are  now 
becoming  major  productivity 
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enhancement  tools  for  data 
processing  organizations.  (Clancy, 
1989)  CASE  tools  are  currently 
available  for  a  wide  variety  of 
computer  systems,  including 
inexpensive  minicomputers  such  as 
IBM  PC-clones.  To  facilitate 
different  kinds  of  software  design, 
the  tools  use  different  design 
methodologies.  As  CASE  tools 
mature  vendors  are  recognizing  the 
need  to  develop  interlocking  tools  to 
cover  design  and  extend  further  into 
the  life  cycle.  The  underlying  ideas 
of  CASE  are  not  new.  They  are  an 
outgrowth  of  the  philosophy  of 
structured  programming  which  has 
its  roots  prior  to  the  1960’s  in  the 
teachings  of  Niklaus  Wirth  and 
Edsger  Dijkstra.  "Structured 

programming  was  the  first  effort  at 
applying  to  software  the  principle  of 
modularity  and  the  use  of 
independent,  reusable  building 
blocks,  each  of  which  performs  a 
single  function."  (EDN,  July  23, 
1990,  225)  This  principle  was 

already  appearing  in  the  construction 
of  bus-based  hardware. 

The  available  CASE  tools  fall  into 
two  basic  categories-first,  tools  that 
clearly  define  what  a  proposed 
software  system  will  do;  second, 
tools  that  specify  the  actual  software 
modules  for  fulfilling  these 
requirements.  The  modules  can  be 
converted  to  high-level  codes. 
These  two  types  of  tools  provide  for 
consistency  among  different  sections 
of  a  system  regardless  of  how  many 
people  are  developing  the  system. 
CASE  software  vendors  adhere  to  a 
number  of  different  schools  of 
thought  on  how  to  implement  these 
ideas. 


Many  of  today’s  CASE  tools  offer 
you  a  choice  among  several  design 
methodologies  for  two  main  reasons. 
First,  such  flexibility  means  that  you 
will  not  necessarily  have  to  learn  a 
new  methodology  to  be  able  to  use 
a  particular  tool.  Second,  if  you  are 
already  familiar  with  several 
methodologies  and  their  notations, 
you  can  apply  the  methodology  that 
best  suits  your  problem. 

CONCEPTS  AND 
MISCONCEPTIONS  OF  CASE 

Underlying  CASE  is  the  concept  of 
software  engineering  which  in  turn  is 
based  on  generally  accepted 
engineering  principles  and  practice. 
These  principles  emphasize 
formalism,  standardized  design, 
planning,  and  control  in  the 
developmental  process.  In  addition 
to  these  basic  engineering  concepts, 
software  engineering  employs  the 
application  of  design  skills,  good 
management  practice,  computer 
science,  and  mathematical  formalism 
to  the  various  facets  of  the  software 
life  cycle  such  as  specification, 
design,  verification,  documentation, 
testing  and  maintenance.  (Necco,  et 
al,  1989) 

It  is  critical  to  recognize  that  these 
techniques  are  performed  within  the 
context  of  a  controlled,  managed 
environment  that  promotes 
productivity,  quality,  and  efficiency 
both  in  the  short  and  long  run. 
Integration  of  the  related  elements 
of  management,  planning,  and 
development  is  also  instrumental  in 
software  engineering.  Problems  are 
dealt  within  the  bounds  t)f  rational 
holistic  approach  as  oppo.sed  to  a 
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piecemeal  or  reactive  approach.  By 
definition,  any  use  of  CASE 
necessitates  adoption  of  this  view. 

Although  the  acronym  is  now  widely 
accepted,  CASE  does  not  yet  have  a 
simple  industry-wide  definition. 
CASE  tools  have  been  defined  as 
providing  a  means  of 
standardization,  discipline,  and 
automation  to  systems  analysis  and 
application  development.  CASE  is  a 
combination  of  procedures,  methods, 
and  software  tools  that  treat  the 
entire  application  development 
process  as  a  system  that  can  be 
automated.  These  same  principles 

that  can  be  applied  to  an 

application  system  can  be  applied  to 
managing  the  computer-based 
training  design  and  development 
process.  (Clancy,  1989) 

Generally  speaking,  software 

engineering  is  a  means  by  whic’’  an 
organization  can  choose  an 
appropriate  methodology  for 
developing  information  systems 

spanning  the  range  of  the  software 

life  cycle  from  the  strategic  planning 
of  systems  to  the  enhancement  of 
existing  applications.  It  can  be 

defined  as  "a  series  of  orderly, 
interrelated  activities  resulting  in  the 
successful  completion,  delivery,  and 
support  of  an  information  system- 
including  programs,  operating 
procedures  and  documentation." 

(Case,  1986)  CASE  is  the 

technology  that  allows  for 
automation  of  certain  or  potentially 
all  endeavors  that  are  undertaken 
within  this  framework. 

The  application  of  CASE  to 
software  engineering  is  directed  at 


achieving  several  main  objectives.  It 
is  geared  toward  increasing  the 
productivity  of  all  individuals 
involved  in  the  engineering  process 
including  systems  analysts, 
programmer,  and  MIS  managers.  It 
enhances  the  quality  of  the  final 
software  product  and  intermediate 
deliverables  in  the  development 
process.  Moreover,  CASE  provides 
for  more  effective  management 
control  over  the  development 
process. 

Basically,  a  CASE  tool  should 
provide  for  a  computerized 
mechanism  for  one  or  all  of  the 
activities  commonly  associated  with 
software  development.  CASE  tools 
that  focus  on  the  earlier  stages  of 
system  development  (e.g.,  planning, 
cost  appraisal,  structured  design)  are 
referred  to  as  front-end  products. 
Those  tools  that  assist  in  the  later 
stages  of  development  (e.g.,  code 
generation,  report  writing)  are 
known  as  back-end.  CASE  products 
that  offer  an  integration  of  CASE 
tools  affecting  all  stages  of  the  life 
cycle  have  been  named  Integrated 
Project  Support  Environments 
(IPSEs)  or,  alternatively.  Integrated 
Computer  Software  Engineering 
(ICASE).  Regardless  of  which  type 
of  CASE  a  company  chooses, 
emphasis  must  be  placed  on  the 
comprehensive  approach  to  systems 
development  and  maintenance  that 
software  engineering  theory  and 
practice  requires.  As  CASE  tools 
develop  and  mature,  these  tools  will 
extend  beyond  design  and  penetrate 
deeper  into  the  life  cycle. 

A  CASE  tool  (or  part  of  a 
comprehensive,  fully  integrated 
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IPSE)  can  also  be  classified  in 
accordance  to  the  general  type  of 
activity  that  it  supports..  There  are 
three  basic  activities  that  CASE 
tools  can  be  used  for:  (1)  strategic 
management  and  planning,  (2) 
systems  analysis/design  and  (3) 
programming  (most  commonly  code 
generation  driven  by  a  structured 
specification  language)  and 
programming  maintenance.  Michael 
Gibson,  Charles  Snyder  and  R.  Kelly 
Rainer  have  designated  CASE 
support  of  these  three  activity 
groupings  as  ’Upper  CASE’,  ’Middle 
CASE’,  and  ’Lower  CASE’, 
respectively.  (Gibson,  et  al.  May 
1989) 

As  might  be  expected  given  the  high 
level  of  confusion  surrounding  CASE 
technology,  there  are  many 
misconceptions  concerning  what  it  is 
and  what  it  can  do.  Unfortunately, 
the  lack  of  understanding  of  CASE 
tends  to  promote  reluctance  on  the 
part  of  MIS  to  adopt  it  and  can 
cause  misjudgments  to  be  made 
concerning  its  costs  and  benefits. 

One  of  the  more  notable 
misconceptions  is  that  CASE  is  a 
replacement  of  fourth  generation 
languages.  As  noted  above,  CASE 
is  available  for  use  in  areas  other 
than  programming.  When  it  is  used 
as  a  programming  aid,  system 
specifications  are  entered  by  the 
user  and  subsequently  converted  into 
either  fourth  or  third  generation 
code  by  the  CASE  system  itself. 
Although  sometimes  referred  to  as  a 
fifth  generation  language,  CASE  is 
not  a  substitute  for  existing 
languages. 


Whereas  CASE  provides  the  greater 
ability  to  track  project  progress  and 
to  develop  strategies  and  plans,  it 
does  not  replace  the  vital  function 
of  leadership  and  direction  on  the 
part  of  management.  A  structured 
environment  is  still  necessary. 
CASE  can  enhance  effective 
management  control,  but  does  not 
eliminate  the  need  for  it.  If 
anything,  CASE  requires  a  more 
disciplined  set  of  procedures  and 
standards  than  traditionally  accepted. 
(Gibson,  1989,  14) 

There  is  a  tendency  to  believe  that 
CASE  eliminates  the  systems  analyst. 
What  CASE  does  is  to  give  the 
analyst  a  set  of  tools  from  which  to 
analyze  and  design  systems  and  their 
specifications.  It  alleviates 

cumbersome  paperwork  and  makes 
designs  more  malleable  and  can 
assist  the  systems  analyst  in 
responding  more  efficiently  to 
unforeseen  changes  in  design 
resulting  from  modifications  in 
applications.  The  role  of  the 
systems  analysts  in  the 
developmental  process  will  be  as 
central  (if  not  more  so)  in  the  fully 
integrated  CASE  environment  than 
in  non-CASE  situations. 

The  perception  that  the  backlog  of 
DP  requests  can  once  and  for  all  be 
ended  by  employing  CASE  tools  is 
erroneous.  Even  though  CASE  has 
the  ability  to  deliver  reliable  systems 
and  satisfy  maintenance  requests 
more  efficiently,  the  sheer  volume  of 
increasing  requests  will  not  eliminate 
existing  backlogs.  Therefore,  it 
would  be  overly  optimistic  for  an 
organization  to  expect  the  backlog 
of  developmental  or  maintenance 
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requests  to  diminish  substantially 
with  a  fully  integrated  CASE  system. 

Some  clarification  is  also  needed 
when  applying  the  terms  CASE  and 
ICASE.  They  are  not  one  and  the 
same  as  is  sometimes  thought. 
Integrated  CASE  (ICASE) 
encompasses  all  phases  of  software 
engineering,  and  more  importantly, 
is  the  vehicle  by  which  specifications 
are  transported  across  these  phases. 
Applications  are  formulated  in  the 
planning  function  in  terms  of  general 
specifications  of  application  systems 
to  be  developed.  "Corporate  IS 
planning  specifications  are  translated 
into  A  &  D  (analysis  and  design) 
specifications  for  the  applications. 
Systems  analysts  use  middle  CASE 
to  add  to  these  A  &  D 
specifications  to  more 
comprehensively  describe  the 
application.  The  design 

specifications  are  translated  into  the 
development  specifications  for 
programs  within  the  application  and 
necessary  end  user  documentation. 
Systems  developers  (then)  add  to 
these  specifications  to  provide  a 
more  comprehensive  set  of 
development  specifications.  Lower 
CASE  systems  then  use  these 
specifications  to  generate  the 
programs  within  the  application  and 
accompanying  end  user 
documentation."  (Gibson,  et  al,  1989, 
12) 

CASE,  as  opposed  to  ICASE. 
involves  the  automation  of  one  or 
more  tasks  or  set  of  functions  that 
define  one  of  the  aforementioned 
phases.  They  operate  independently, 
at  least  in  the  .sense  that  they  do 
not  directly  interface  with  other 


components  except  through  manual 
efforts.  A  CASE  tool  that  operates 
in  a  non-integrated  situation  is  called 
a  stand-alone  CASE.  The  trend, 
however,  is  toward  consolidation  of 
CASE  tools  into  a  total 
comprehensive  environment  (i.e., 
ICASE)  and  away  from  segregated 
tools. 

It  should  be  recognized  that  CASE 
will  not  result  in  an  immediate 
increase  in  productivity.  There  is  a 
significant  amount  of  overhead 
involved  in  starting  up  a  CASE 
system.  Documentation  and 

specifications  must  be  manually 
entered  into  a  newly  introduced 
CASE  system.  Determining  the 
location  of  such  documentation  can 
be  time  consuming.  In  some  cases, 
the  documentation  may  not  exist  at 
all  or  may  be  scattered  in  many 
places  and  storage  types  from 
manual  forms  and  documents  to 
word  processing  files,  and  so  on. 
Not  until  this  type  of  start-up  cost  is 
completed  can  productivity  gains 
from  CASE  even  begin  to  be 
realized.  Productivity  will  generally 
be  increased  over  the  long  run;  at 
that  point  the  benefits  may  increase 
geometrically  if  CASE  is  employed 
properly. 

ANALYSIS  AND  EXPECTATIONS 
OF  CASE 

Before  the  current  status  of  CASE 
is  analyzed,  CASE  must  be  further 
defined.  Simply  stated,  CASE 
technology  is  the  collection  of  tools 
designed  to  assist  the  system 
professional  in  his  duties.  These 
duties  include  all  phases  of  the 
systems  life  cycle:  analysis,  design. 
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implementation,  and  maintenance. 
(McClure  and  Ambrosio,  June  1989, 
33-42)  A  CASE  tool  can  be 
designed  to  help  out  in  one,  several, 
or  all  of  these  areas.  Therefore, 
CASE  tools  can  have  a  wide  variety 
of  functions,  from  simple  data-flow 
diagram  plotters  to  complex  system 
code  generations. 

In  this  analysis,  users’  expectations 
of  CASE  systems  highlighting  what 
they  feel  are  the  most  important 
factors  in  a  CASE  system  will  be 
addressed  as  well  as  a  look  at  the 
current  benefits  and  drawbacks  of 
current  CASE  technology.  The 
analysis  will  be  concluded  with  an 
evaluation  of  how  CASE  technology 
meets  up  with  users’  needs  and 

expectations. 

In  a  recent  survey  of  about  800 
computer  installations,  the  following 
were  listed  as  the  eight  most 
important  CASE  tool  features. 
Several  conclusions  can  be  reached 
from  this  data.  First  and  foremost, 
users  are  concerned  with  the 

development  of  viable  systems. 
Second,  users  desire  tools  that  either 
deal  with  all  aspects  of  the  systems 
life  cycle  or  work  with  other  tools  to 
create  a  total  package.  Finally, 
users  are  concerned  with  the 

functioning  of  the  tools,  desiring 
packages  that  utilize  a  methodology, 
have  prototyping  ability,  and  are 
well  supported  by  the  developer  or 
vendor.  The  list  of  the  eight  most 
important  CASE  tool  features 
follows:  (McClure  and  Ambrosio,  9, 
3,  1989,  33) 

Checks  Logic  96.3% 

Supports  Full  Life  Cycle  77.5% 


Fully  Integrated  74.0% 

Prototyping  Capab.  72.7% 
Connectivity  71.0% 

Contains  a  Repository  68.5% 
SupportA’raining  67.3% 

Uses  Standard 

Technique/Methodology  66.4% 
BENEFITS 

One  of  the  primary  benefits  of 
CASE  technology  is  that  it  brings 
about  the  use  of  a  standard 
methodology  to  all  systems  projects. 
When  a  systems  professional  uces  a 
CASE  tool  or  a  set  of  CASE  tools 
for  his  projects,  all  of  those  projects 
are  developed  using  the  same 
methodology  as  the  tools  will  not 
allow  steps  to  be  skipped.  (McClure 
and  Ambrosio,  March  1989,  34,  41) 
This  kind  of  forced  methodology 
results  in  a  consistent  product. 

There  is  one  major  warning  about 
this  aspect  of  CASE  tools:  one 
must  be  very  careful  to  match  the 
methodologies  employed  by  the 
desired  CASE  tools  with  the 
strategic  direction  of  the  information 
systems  department  and  the  overall 
information  needs  of  the 
organization.  (McClure  and 
Ambrosio,  June  1989,  40.)  Many 
failures  in  CASE  implementations 
have  resulted  from  a  failure  to 
provide  such  a  match.  These 
mismatches  can  have  a  wide  variety 
of  results  from  systems  that  users 
feel  uncomfortable  with  to  systems 
that  do  not  work  at  all. 

There  are  several  choices  of 
methodologies  that  are  currently 
available.  One  of  the  favorites  is 
the  traditional  Process-Oriented 
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methodology  as  it  is  one  that  most 
systems  professionals  are  familiar 
with  and  have  had  experience  using. 
(Bailin,  S.  May  1989,  608) 

However,  there  are  two  new 
methodologies  that  are  gaining  in 
popularity  for  many  applications: 
Information  Engineering  and 
Prototyping.  Information 

Engineering  is  one  example  of  a 
data-oriented  approach  to  systems 
development.  In  this  methodology, 
data  design  takes  precedence  over 
procedure  design.  First,  logical 
models  of  the  data  used  by  the 
organization  are  developed.  Then, 
individual  application  systems  are 
developed  using  this  model.  In  this 
manner,  applications  can  be  better 
integrated  and  data  sharing  can  be 
better  controlled.  (McClure  and 
Ambrosio,  June  1989,  34,  41) 

Prototyping  is  the  development  of 
an  executable  system  that  can  model 
the  functioning  of  the  desired 
system.  (Tanik  and  Yeh,  May  1989, 
9-10)  This  methodology  is  an 
iterative  process  with  the  prototype 
being  continually  adjusted  and 
refined  in  response  to  the  user’s 
reactions.  This  process  is  continued 
until  the  model  is  found  to  be 
adequate-  to  serve  as  the  actual 
system.  (McClure  and  Ambrosio, 
June  1989,  41)  Prototyping  is 

particularly  useful  in  the 
development  of  systems  that  have  a 
high  level  of  user  interfacing.  (Rents, 
et  al.  May  1989,  59) 

Another  major  benefit  of  CASE 
technology  is  the  increase  in  the 
quality  of  code  developed  for  a 
system.  (Bouldin,  August  1989,  30- 
39)  One  of  the  primary  reasons  for 


this  is  the  automatic  error  checking 
features  that  are  present  in  most 
code-generating  tools.  This 

increased  quality  results  in  fewer 
abends,  fewer  change  requests  by 
users,  and  fewer  mistakes  caught 
late  in  the  cycle  which  can  lead  to 
enormous  benefits.  Another  aspect 
of  this  benefit  is  an  increase  in  code 
maintainability.  (McClure  and 
Ambrosio,  June  1989,  41)  Because 
CASE  generated  code  is  of  a  higher 
quality  than  traditionally  generated 
code,  it  is  easier  to  maintain.  This 
is  also  due  to  the  fact  that  CASE  is 
generally  better  structured  which 
makes  it  easier  for  the  systems 
professional  to  look  at  a  block  of 
code  and  determine  what  it  is  doing. 

One  possible  downside  of  this 
benefit  is  that  although  code  is 
generally  being  generated  better,  it 
is  not  necessarily  being  generated 
faster.  However,  this  is  not  seen  as 
being  a  drawback  of  CASE  tools. 
According  to  Pam  Fox  of  Lincoln 
National  Corporation,  "(we  are  not) 
as  concerned  with  productivity  as  we 
are  with  quality.  We  want  to  give 
(developers)  tools  that  will  help 
them  do  their  jobs  better,  not 
faster." 

Another  benefit  of  CASE  tools  is 
the  development  code  that  will  be 
reusable;  i.e.,  code  that  can  remain 
fundamentally  unchanged  each  time 
the  system  is  modified  or 
restructured.  Not  only  the  code 
becomes  reusable  but  the  existing 
software  offers  a  way  to  achieve 
lower  development  costs  as  well  as 
quality  for  these  new  software 
products.  Many  believe  that  the 
promise  CASE  tools  have  in  this 


area  is  far  brighter  than  the  actual 
progress  achieved  to  date. 
According  to  Adrian  McManus  of 
American  Express,  'The  real 
benefits  will  come  with  truly 
reusable  code  that  can  be  saved  at  a 
higher,  more  abstract  level-entity 
relationship  models  and  data  flow 
di?7rams."  (McClure  and  Ambrosio, 
June  1989,  41)  However,  code 
generated  from  a  CASE  system  still 
has  a  greater  potential  for  reusability 
than  traditionally  generated  code 
because  of  its  higher  quality  and 
easier  maintainability  so  this  can  still 
be  listed  as  a  benefit. 

DRAWBACKS 

The  major  drawback  of  CASE 
technology  is  the  lack  of  industry 
standards  for  the  interfacing  of 
various  CASE  tools.  This  lack  of  a 
consistent  applications  interface  has 
hurt  CASE  as  it  makes  it  difficult 
for  the  systems  professional  to  use  a 
group  of  tools  from  different 
vendors  to  develop  a  single  system. 
According  to  McManus,  "I  don’t 
know  anyone  .  .  .  who  is  completely 
satisfied  with  any  one  tool.  So,  you 
have  to  buy  different  vendor’s  tools 
for  different  phases.  Vendors  will 
have  to  come  to  grips  with 
(interfacing)  their  products." 
(McClure  and  Ambrosio,  June  1989, 
40)  This  lack  of  interfaceability  puts 
the  systems  professional  in  a 
dilemma.  Either  he  uses  a  single 
total  life  cycle  tool  for  a 
development  that  does  not  do 
everything  the  way  he  would  like  it 
to  or  he  uses  the  best  tools  for  each 
aspect  and  had  to  deal  with  trying 
to  integrate  the  tools.  Clearly  this  :s 
an  issue  that  must  be  resolved  in 


order  to  make  CASE  a  truly  useful 
technology.  One  notable  exception 
to  the  existing  situation  is 
KnowledgeWare’s  "Application 
Development  Workbench  (ADW)" 
which  is  a  comprehensive  collection 
of  CASE  tools  integrated  through  an 
Artificial  Intelligence  (Al)-based 
repository.  (Application 
Development  Workbench,  1991,  2) 

Another  drawback  of  CASE  is  that 
there  is  a  significant  learning  curve 
inherent  in  the  technology. 
Typically,  there  is  a  two-project 
learning  curve  where  productivity  is 
very  low.  Only  after  these  first 
projects  does  CASE  technology 
begin  to  approach  the  productivity 
of  the  previously  employed 
technique.  (McClure  and  Ambrosio, 
June  1989,  41)  However,  despite 
the  learning  curve,  CASE  tools  still 
produce  a  product  of  high  quality 
and  reliability  right  from  the  start. 

A  third  drawback  of  CASE 
technology  is  the  cost  involved  with 
a  CASE  implementation.  To  start 
with,  there  is  the  initial  cost  of  the 
CASE  software  and  the  hardware  to 
run  it  on.  This  is  not  all  the  costs, 
in  addition,  there  are  the  training 
costs  associated  with  the  CASE 
technology  which  have  been 
estimated  as  being  about  three  times 
as  much  as  the  cost  of  the  software. 
(McClure  and  Ambrosio,  June  1989, 
43) 

Because  of  this  significant  capital 
requirement,  CASE  tools  are 
frequently  avoided  by  organizations 
that  look  only  to  immediate  benefits, 
rather  than  long-term  returns.  As 
the  majority  of  CASE  benefits  occur 


253 


in  the  long  run,  the  technology  tends 
to  look  bad  when  evaluated  in  this 
manner.  Only  organizations  that 
understand  the  future  value  of 
CASE  technology  tend  to  overlook 
this  drawback. 

EVALUATION 

In  an  evaluation  of  CASE  tool 
performance,  one  must  first  look  at 
how  users  tend  to  benchmark  CASE 
effectiveness.  In  a  recent  survey, 
experienced  CASE  users  ranked 
three  decision  criteria  with  the 
following  results;  (Bouldin,  August 
1989,  31) 

Primary  Criteria: 


Time  Savings:  47.5% 

Quality  Improvements  39.2% 

Financial  Savings  13.3% 

Secondary  Criteria: 

Time  Savings:  38.8% 

Quality  Improvements  36.9% 

Financial  Savings  24.6% 


Tertiary  Criteria; 

Time  Savings:  13.8% 

Quality  Improvement  21.4% 

Financial  Savings  61.9% 

When  viewed  in  this  light.  CASE 
technology  looks  pretty  good.  CASE 
technology  offers  significant  time 
savings  once  the  learning  curve  has 
been  overcome  and  it  has  a  proven 
record  in  the  area  of  code  quality 
improvement.  Although  CASE 
technology’s  financial  savings  can  be 
in  doubt,  particularly  in  the  short 


run,  this  is  not  a  significant  problem 
as  this  aspect  is  not  viewed  as  being 
as  important  as  the  other  two. 

Another  way  to  evaluate  CASE 
technology  is  to  look  at  it  in  terms 
of  the  eight  important  features  that 
were  previously  discussed.  In  this 
manner  one  can  evaluate  how  CASE 
is  meeting  users’  needs  rather  than 
how  users  evaluate  CASE. 

In  terms  of  four  of  these  major 
features,  CASE  is  currently 
performing  very  well.  As  was 
discussed  in  the  benefits  section, 
CASE  tools  generally  check  logic 
providing  the  user  with  high  quality 
code.  CASE  tools  also  can  have 
very  good  support  and  training,  but 
the  user  is  going  to  have  to  pay  for 
these  features.  Because  of  the 
nature  of  their  design,  proper  use  of 
CASE  tools  will  provide  the  user 
with  a  standard  methodology  for  all 
projects,  and,  if  the  proper 
methodology  is  selected,  CASE  tools 
can  provide  excellent  prototyping 
capabilities. 

However,  in  terms  of  three  of  these 
major  features,  CASE  is  currently 
not  performing  up  to  the  users’ 
expectations.  As  was  discussed  ii; 
the  drawbacks  section,  although 
many  CASE  tools  support  the  full 
development  life  cycle,  the 
performance  of  individual  elements 
of  these  tools  is  usually  not  up  to 
the  users’  standards.  Also,  with  the 
exception  of  vendor  development 
such  as  KnowledgeWare’s  ADW 
cited  previously,  CASE  tools  are 
poorly  integrated  and  vendors 
currently  offer  little  connectivity  with 
other  vendors’  tools. 
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The  last  of  these  major  features,  the 
utilization  of  a  repository,  really 
does  not  fall  into  this  evaluation.  If 
a  repository  is  important  to  an 
individual  user,  he  should  limit  the 
decision  set  to  CASE  tools  that  have 
this  capability.  Otherwise,  a  user  is 
free  to  choose  from  all  the  available 
tools. 

To  conclude  this  evaluation,  it 
appears  that  CASE  tools  are 
meeting  the  users’  expectations  and 
requirements  fairly  well,  but  there 
still  is  room  for  significant 
improvement.  CASE  tools  are  very 
strong  in  many  of  the  technical 
aspects  of  the  systems  development 
process.  However,  they  are  still  very 
weak  in  one  major  area  of 
importance  to  the  user;  integration 
and  connectivity.  In  order  for 
CASE  tools  to  be  truly  viable  in  the 
future,  it  is  very  important  for  this 
area  to  be  addressed. 


FUTURE  OF  CASE 

As  has  just  been  discussed,  CASE, 
although  an  impressive  technology, 
still  has  some  problems  to  be 
worked  out  in  order  to  make  it  truly 
invaluable  to  the  systems 
professional.  These  problems  are 
not  necessarily  insurmountable,  but 
they  do  represent  some  of  the  major 
barriers  against  universal  CASE 
adoption.  However,  if  these 
prob'ems  are  solved,  there  are  many 
areas  that  will  be  of  future 
consideration  to  the  CASE 
technologist. 

In  this  section,  we  will  go  over  some 
of  the.se  areas:  Prototyping,  Expert 


Systems,  Project  Management,  and 
the  trend  to  go  Back  to  the  Basics. 
They  represent  s^iiie  new 
applications  of  CASE  technology, 
some  expansions  and  modifications 
cf  existing  applications  and  one 
possible  threat  to  CASE. 

Prototyping.  As  previously 

mentioned,  prototyping  is  the 
process  of  developing  a  scaled-down 
version  of  a  system  to  use  in 
building  a  full-scale  system.  (Tanik 
and  Yeh,  May  1989,  9)  Although 
prototyping  is  a  currently  utilized 
methodology  in  CASE  technology,  it 
still  is  an  area  of  massive  potential 
for  the  future.  There  are  three 
areas  of  prototyping  that  are 
believed  to  nave  a  large  growth 
potential:  Software  Evolution, 

Object-oriented  CAD,  and  Software 
Storming. 

Software  evolution  refers  to  all 
activities  that  alter  a  software 
system,  including  requirements 
changes,  performance  changes,  and 
repair.  Essentially,  it  is  systems 
maintenance  in  it  broadest  scope.  It 
is  a  very  important  consideration  in 
systems  development,  as  software 
evolution  accounts  for  more  than 
half  of  the  total  software  cost.  '.^Luqi, 
May  1989,  14)  It  is  hoped  that  this 
would  result  in  a  much  lowered 
software  evolution  cost  becaii.se  of 
this  ease  of  maintenance. 

Object-oriented  CAD  is  truly  a 
hybrid  technology  as  it  represents 
the  application  of  CASE  technology 
to  the  field  of  circuit  design.  It 
follows  CASE  technology  in  that  it 
allows  designers  to  gradurlly  refine  a 
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subset  of  operations  rather  than 
developing  a  complete  application. 
(Gupta,  et  al,  May  1989,  28)  It  also 
involves  the  development  of 
improved  software  tools  for 
hardware  testing.  (Davidson,  April 
1989,  12)  It  will  be  of  increasing 
interest  in  the  future  as  it  allows  for 
a  convergence  of  knowledge  between 
software  and  hardware  developers. 

Software  storming  is  another  hybrid 
technology  as  it  represents  the 
combination  of  knowledge 
engineering  (the  brainstorming 
problem  solving  technique)  and 
systems  development  technology. 
Using  this  method,  a  team  of 
researchers  developed  in  four 
months  a  software  prototype  with 
significantly  more  functionality  than 
a  conventional  prototype  that  took 
two  years  to  develop.  (Jordan,  et  al. 
May  1989,  39)  This  area  represents 
a  major  breakthrough  in  shortening 
the  amount  of  time  required  to 
develop  a  system. 

However,  there  are  five 
considerations  to  be  looked  at 
before  attempting  to  implement  such 
a  technique.  I^ow  the  problem: 
software  storming  is  not  appropriate 
for  ill-defined  or  broadly  focused 
problems.  Know  the  team: 

Storming  requires  tremendous 
cooperation  between  team  members 
as  the  team  must  quickly  adapt  to 
changing  roles  during  the  storm. 
Know  the  tools:  if  unfamiliar  tools 
are  being  used,  time  will  have  to  be 
allotted  to  develop  expertise  in 
them.  Know  the  data:  every  aspect 
of  the  data  environment  should  be 
understood.  Finally,  document  in 
video  as  videotapes  of  the  storming 


sessions  are  particularly  useful  in 
catching  the  developed  techniques. 
(Jordan,  et  al.  May  1989,  47) 

Clearly  from  the  above 
considerations,  software  storming  is 
not  a  universal  solution  to  systems 
development  problems.  However, 
its  massive  potential  for  high 
productivity  in  certain  situations 
makes  it  something  that  should  be 
considered  for  the  future.  Perhaps 
further  research  in  this  area  will 
result  in  techniques  that  are  more 
readily  applicable  to  common 
situations  and  still  give  improved 
results. 

Expert  Systems.  Expert  systems  are 
another  area  that  has  potential  to 
reap  enormous  benefits  from  CASE 
technology.  Traditionally,  many  of 
the  most  successful  expert  systems 
were  developed  by  a  trial  and  error 
methodology.  It  is  felt  that  taking  a 
more  structured  approach  in  the 
development  of  expert  systems  will 
allow  the  designers  to  avoid  the 
mistakes  of  the  past.  It  is  hoped 
that  CASE  technology  will  become 
this  structure.  One  of  the  major 
reasons  that  this  would  be  possible 
in  the  area  of  expert  systems  is  that 
50%  to  75%  of  the  work  in 
designing  a  system  is  traditional 
systems  development.  (Blackman, 
February  1990,  27,  31)  Perhaps  the 
first  Expert  System  connection 
development  of  intelligent  user 
interfaces  for  decision  support  tools, 
such  as  databases  and  decision 
models,  is  in  analogical  reasoning 
systems  or  model  reasoning.  This  is 
CASE-based  reasoning  ongoing 
research  in  intelligent  problem 
formulation.  (Blanning.  October 


1990) 

Project  Management  One  of  the 
biggest  problem  areas  in  MIS  is 
project  management.  There  are 
horror  stories  of  numerous  incidents 
of  runaway  development  projects 
which  have  adversely  affected 
budgets,  reputations,  and  even 
competitive  capabilities  of  the 
companies  involved.  (Levine,  March 
1989,  32)  The  integration  of  project 
management  techniques  into  CASE 
methodologies  would  help  control 
many  of  these  projects. 

It  is  becoming  increasingly  clear  that 
management  wants  software 
developers  to  have  the  same  sort  of 
accountability  that  other  functional 
areas  have  for  their  projects.  They 
are  tired  of  treating  systems 
development  as  an  art  and  want  to 
see  some  traditional  management 
techniques  applied  to  it.  It  is  felt 
that  the  application  of  such 
techniques  would  have  the  following 
results:  (Jordan,  et  al.  May  1989,  34) 

1)  Risk  reduction  in  planning 

2)  Management  acceptance 

3)  Early  performance 

analysis 

4)  Control  over  developers’ 

activities 

5)  Control  over  configuration 

of  modules 

Software  developers  also  realize  the 
need  for  controls  over  the 
development  process.  It  is  felt  that 
the  application  of  project 
management  techniques  would 
improve  the  overall  coordination  and 
performance  of  a  development  team. 
A  survey  of  developers  revealed  that 


they  desire  many  of  the  traditional 
features  of  project  management, 
including:  (Gupta,  et  al.  May  1989, 
32-33) 

o  Critical  path  scheduling 
o  Cost  and  schedule  integration 
o  Time-phased  quantity  and 
budget  reporting 
o  Resource  summarizations 
o  "What  if  analysis  capability 

This  is  an  area  that  can  be 
addressed  by  CASE  technology. 
CASE,  because  of  its  methodological 
basis,  already  contains  much  of  the 
structural  design  of  a  project 
management  system.  It  is  just  a 
matter  of  merging  these  two  areas 
to  make  a  simple  consolidated 
product  that  can  handle  total  life 
cycle  design  and  project 
management.  Currently  there  are 
many  systems  that  purport  to  offer 
CASE  project  management  (Levine, 
March  1989,  35-38)  However, 

because  of  the  current  lack  of 
integration  between  CASE  tools, 
most  of  these  packages  cannot 
satisfy  the  requirements  of  both  the 
developers  and  management.  It  is 
for  this  reason  that  project 
management  is  considered  an  area 
for  future  development  in  CASE.  It 
is  hoped  that  once  the  integration 
problems  are  solved,  we  will  be  able 
to  develop  truly  useful  CASE  tools 
that  will  satisfy  everyone. 

Back  to  Basics.  One  trend  that 
CASE  technology  must  look  out  for 
is  the  desire  on  the  part  of  both 
systems  professionals  and  users  to 
move  back  to  a  more  simplistic 
process  with  greater  user 
involvement.  (Synoradzki,  September 
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1988,  30)  This  trend,  if  it  is 
realized,  could  greatly  affect  the 
future  of  CASE  technology.  It  is 
felt  that  developers  of  CASE  tools 
should  keep  this  in  mind  as  they 
make  the  tools  of  the  future. 

Perhaps  the  development  of  more 
end-user  oriented  CASE  tools  is  in 
order.  A  similar  development 
occurred  when  spreadsheet  programs 
brought  significant  computing  power 
to  the  hands  of  the  end  user  by 
simplifying  the  process  of  generating 
financial  and  other  number-intensive 
applications.  This,  however,  could 
be  considered  a  good  possibility  as 
CASE  tools  tend  to  be  very  complex 
instruments,  similar  to  many 
accounting  programs  that  were  very 
complex  until  the  development  of 
Lotus  1-2-3. 

CONCLUSION 

In  summary,  since  the  arrival  of 
improved  microcomputers  in  the 
mid-198()’s,  CASE  tools  have 
become  the  dominant  trend  in 
.systems  development.  They 

encompass  many  of  the  other  trends 
such  as  prototyping  and  code 
generators.  CASE  tools  also  come 
in  a  wide  variety  of  capabilities  from 
simple  data  flow  diagram  plotters  to 
total  life  cycle  packages. 

User  expectations  of  CASE  tools  are 
very  high  and  are  currently  being 
only  partially  met.  Benefits  include 
standard  methodology,  increa.se  in 
code  quality,  and  the  development 
of  reusable  code.  Drawbacks  of 
CASE  include  lack  of  integration  of 
tools,  a  high  learning  curve,  and 
high  costs. 


CASE  technology  is  facing  many 
opportunities  for  further 
development  in  the  future.  The 
decade  of  the  ’90’s  can  be  viewed  as 
the  beginning  of  the  third  generation 
of  CASE.  There  are  many  areas 
that  can  be  further  developed, 
including  Prototyping,  Expert 
Systems,  Project  Management,  and 
Repository  Products.  However, 
CASE  developers  must  also  be 
aware  of  the  possible  trend  of 
bringing  more  computer  power 
directly  to  the  end  user. 

Organizations  around  the  world  are 
now  using  CASE.  CASE  tools  have 
been  around  for  more  than  five 
years  with  more  than  a  dozen 
companies  in  the  CASE  business. 
We  now  have  established  guidelines 
for  evaluating  CASE  tools  and  know 
the  right  questions  to  ask  in  advance 
for  determining  wise  choices. 

In  conclusion,  CASE  is  a  technology 
that  is  truly  at  a  critical  point  in  its 
development.  It  has  had  some 
degree  of  success,  but  it  still  has 
much  potential  that  it  has  to  live  up 
to.  As  CASE  technology  matures,  it 
is  expected  that  it  will  become  an 
even  more  dominant  trend  in 
software  development  of  new 
systems  and  a  positive  force  in 
renewing  the  effectiveness  of  aging 
systems.  Now  that  the  CASE  vision 
has  been  created,  its  universal 
acceptance  requires  selling  it  through 
education. 
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A  COMPREHENSIVE  ENGINEERING  APPROACH  FOR 
INTELLIGENT  TRAINING  SYSTEMS 


Wendy  Sallman 
Robert  Jancoski 

FMC,  in  cooperation  with  Virginia  Polytechnic  Institute,  developed  the  Hoist  Trainer  for  use 
with  the  FMC  manufactured  Mark  45  naval  gun.  The  Mark  45  is  a  complex  weapon  system  that 
consists  of  15  major  assemblies;  one  assembly  being  the  lower  hoist. 

To  learn  the  hydraulic  operation  of  the  lower  hoist,  the  primary  instruction  aid  used  in  the 
classroom  today  is  a  schematic  drawing.  Teaching  the  hydraulic  operation  of  a  weapon  system 
with  a  static  schematic  can  be  difficult  and  time-consuming  because  this  approach  relies  heavily  on 
the  student's  ability  to  conceptualize  operation  s/he  cannot  readily  observe.  To  help  students 
understand  the  hydraulic  operation  of  the  lower  hoist,  FMC  developed  the  Hoist  Trainer. 

The  Hoist  Trainer  is  a  computer-based  training  system  designed  to  teach  students  the  hydraulic  and 
mechanical  operation  of  the  Mark  45  lower  hoist.  The  trainer  features  a  qualitative  model  of  the 
lower  hoist,  a  counterfactual  inference  engine,  and  a  schematic-based  user  simulation.  Computer- 
based  training  offers  the  benefits  of  consistency,  easy  distribution,  and  improved  student 
performance  and  retention. 

The  innovative  technical  concepts  represented  by  the  training  system  are  applicable  to  any 
hydraulic/mechanical  system.  In  addition,  the  system  components  can  extend  into  other  application 
domains  such  as  model-based  diagnostics  or  intelligent  tutor  systems.  This  paper  discusses  the 
Hoist  Trainer  architecture,  lessons  learned  during  tkvelopment,  FMC  developed  software  tools 
that  will  aid  in  the  engineering  of  future  training  systems,  and  the  applicability  of  Hoist  Trainer 
concepts  to  other  applications. 
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training  systems. 
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A  COMPREHENSIVE  ENGINEERING 
APPROACH  FOR 

INTELLIGENT  TRAINING  SYSTEMS 

Wendy  Sallman 

Robert  Jancoski 

Mark  45  Lower  Hoist  Background 
The  Hoist  Trainer  has  been  developed  for  use 
with  the  FMC  manufactured  Mark  45  naval 
gun.  The  Mark  45  is  a  complex  weapon 
system  that  consists  of  15  major  assemblies 
with  over  23,000  parts.  One  major  assembly 
of  the  gun  is  the  lower  hoist.  This  assembly 
functions  as  the  transfer  mechanism  of  the  gun 
that  delivers  ammunition  from  its  storage  room 
to  its  ready  magazine.  The  ammunition  within 
the  lower  hoist  rests  on  pawls  that  are  attached 
to  a  hydraulic  rack.  The  entire  lower  hoist 
transfer  mechanism  is  controlled  by  solenoids 
that  convert  electrical  signals  to  hydraulic 
pressures. 

The  lower  hoist  consists  of  approximately  150 
elements  or  parts.  These  elements  include: 
seven  types  of  pistons,  three  types  of  latches, 
two  types  of  solenoids,  four  types  of  state¬ 
detecting  switches,  a  chain,  a  linkage,  a 
hydraulic  rack,  a  clutch  mechanism,  and 
hundreds  of  hydraulic  pipes. 

To  understand  the  operation  of  the  lower  hoist 
or  any  weapon  system  often  requires  the 
student  to  derive  the  operation  of  the  system, 
using  hand-colored  schematic  drawings.  This 
approach  to  learning  weapon  system  operation 
is  usually  time-consuming  and  difficult  and 
does  not  ensure  depth  or  consistency  of 
training.  To  allow  students  to  systematically 
learn  the  operation  of  the  Mark  45  lower  hoist, 
FMC  developed  the  Hoist  Trainer. 

Hoist  Trainer 

The  Hoist  Trainer  is  a  model-based  training 
system  designed  to  teach  students  the  internal 
operation  of  the  Mark  45  lower  hoist.  The 
Hoist  Trainer  offers  the  benefits  of  computer- 
based  training:  consistency,  easy  distribution, 
and  improved  student  performance  and 
retention. 

The  Hoist  Trainer  simulates  the  ammunition 
load  and  off-load  operations  of  the  lower  hoist. 


Both  processes  involve  six  operational  machine 
cycles: 

1.  engage  coupling 

2.  rack  extend 

3.  drop  engage  coupling 

4.  disengage  coupling 

5.  rack  retract 

6.  drop  disengage  coupling. 

The  Hoist  Trainer  is  deployed  on  the  Texas 
Instruments  microExplorer.  The  system 
architecture  consists  of  three  major 
components,  each  having  specialized,  but 
interrelated  duties:  a  qualitative  model,  a 
counterfactual  inference  engine,  and  a  user 
view  that  includes  a  schematic-based  simulation 
and  an  intuitive  user  interface.  These  system 
components  are  discussed  in  the  following 
paragraphs. 

Qualitative  Model 

The  qualitative  model  of  the  Hoist  Trainer  was 
developed  by  Virginia  Polytechnic  Institute 
(VPI)  and  funded  in  part  and  extended  by 
FMC's  Naval  Systems  Division  [1,2,3].  The 
model  uses  qualitative  physics  to  predict  and 
explain  the  behavior  of  the  lower  hoist  in 
qualitative  terms.  Qualitative  reasoning  is  used 
to  derive  the  normal  operation  of  the  lower 
hoist  from  a  structural  model.  The  lower  hoist 
structural  model  consists  of  elements,  a 
representation  of  element  behavior,  and  the 
connections  between  elements.  Elements 
within  the  Hoist  Trainer  can  be  described  as 
mechanical,  hydraulic,  or  electrical  parts  (e.g. 
pistons,  solenoids,  pipes). 

The  connections  between  the  elements  in  the 
Hoist  Trainer  are  typically  hydraulic  pipes. 
The  connections  are  governed  by  the  principle 
of  locality  and  are  expressed  as  causal 
relationships.  The  principle  of  locality  states 
that  a  single  causal  relationship  can  only 
influence  the  behavior  of  its  neighbors.  No 
one  element  can  have  direct  influence  over  the 
behavior  of  the  entire  machine  [1,2,5]. 

Counterfactual  Inference  Engine 
The  second  component  of  the  Hoist  Trainer  is 
the  counterfactual  inference  engine.  This 
engine  was  developed  by  VPI  and  funded  in 
pan  by  FMC's  Naval  System  Division  [3,4,5]. 
The  counterfactual  inference  engine  operates  on 
the  qualitative  model  to  create  a  current  world 
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or  a  representation  of  the  lower  hoist  operation 
that  is  consistent  at  a  Axed  point  in  time. 

For  example,  assume  the  qualitative  model  and 
the  engine  have  created  a  world  that  represents 
the  engage  coupling  cycle.  In  this  scenario,  the 
world  is  represented  by  the  state  of  lower  hoist 
elements  (the  pistons,  solenoids,  etc.)  at  the 
start  of  the  engage  coupling  cycle  and  the 
causal  relationships  that  connect  the  elements. 

To  cause  the  engage  coupling  cycle  to  be 
advanced  to  the  next  cycle  of  operation,  the 
rack  extend  cycle,  a  counterfactual  is 
introduced.  A  counterfactual  is  a  piece  of 
knowledge,  or  a  fact,  that  contradicts  at  least 
one  believed  state  of  the  current  world.  For  the 
purpose  of  our  example,  assume  the 
counterfactual  represents  a  solenoid  that 
changes  from  the  current  not  energized  state  to 
the  energized  state.  This  solenoid  state,  when 
introduced  into  the  current  world,  distorts  the 
world  and  creates  a  world  that  is  inconsistent. 
The  counterfactual  inference  engine  uses  the 
counterfactual,  "solenoid  energized,"  and 
assumes  that  it  is  true  in  order  to  generate  and 
propagate  all  possible  sets  of  lower  hoist 
machine  states  that  would  explain  or  make  a 
consistent  world.  In  other  words,  the 
counterfactual  inference  engine  works  with  the 
qualitative  model  to  create  a  world  that  makes 
sense  for  the  "solenoid  energized" 
counterfactual.  The  resulting  world  is  again 
consistent  and  represents  the  next  cycle  of 
lower  hoist  operation  -  rack  extend.  The 
representation  of  this  world  is  a  list  of  lower 
hoist  machine  states.  FMC  uses  this  list  of 
information  to  drive  the  schematic-based 
simulation  of  the  Hoist  Trainer. 

User  View 

The  user  view  of  the  Hoist  Trainer  is  an 
important  component  of  the  training  system.  A 
typical  student  does  not  care  about  the 
underlying  technologies  or  the  system 
architecture.  The  student  is  mainly  concerned 
with  how  to  use  and  understand  the  system. 
Using  this  knowledge  of  the  student  or  user, 
FMC  developed  a  user  view  for  the  Hoist 
Trainer  that  consists  of  two  subcomponents:  a 
user  interface  and  a  schematic-based 
simulation. 


User  Interface 

The  user  interface  is  the  student  gateway  to  the 
Hoist  Trainer.  Its  intuitive  design  allows  the 
student  to  experiment  with  the  training  system 
through  self-explanatory,  pull-down  menus 
and  dialog  boxes.  This  approach  to  user 
interface  design  allows  students  to  woiir  at  their 
own  pace  and  to  tailor  the  instruction  to  their 
own  needs.  Individualized  training  allows 
students  to  explore  only  what  they  need  to 
know,  without  repeating  concepts  they  have 
already  mastered  [6]. 

Four  user  menus  were  developed  to  allow  the 
student  to  control  the  execution  of  the  Hoist 
Trainer:  Control,  Mode  of  Operation,  Load 
and  Off-Load. 

The  Control  menu  provides  the  student  with  an 
option  to  take  single  steps,  forward  or 
backward,  through  the  current  cycle.  For 
example,  the  student  could  step  backwards 
through  the  rack  extend  cycle  to  visually  review 
why  a  lower  hoist  element  changed  state.  In 
contrast  to  single  step  control,  the  control  menu 
contains  an  automatic  option  which  allows  the 
student  to  observe  the  mechanical  operation  of 
an  entire  cycle.  Reset  and  quit  menu  selections 
allow  the  student  to  restart  the  simulation  and 
exit  the  application,  respectively. 

The  Mode  of  Operation  menu  allows  the 
student  to  select  and  view  the  execution  of  the 
load  or  off-load  operation  of  the  lower  hoist. 
Depending  upon  the  mode  of  operation 
selected,  the  Load  or  Off-Load  Cycle  menu  is 
enabled. 

The  Load  Cycle  and  the  Off-Load  Cycle 
menus,  are  similar  in  structure.  The  lower 
hoist  operation  for  each  of  these  cycles  includes 
the  same  cycles,  but  the  cycles  occur  in  a 
different  order  The  cycle  menus  let  the  student 
advance  the  model  to  any  cycle  of  operation  to 
experiment  with  the  training  system.  This 
allows  the  student  to  examine  the  operation  of 
the  lower  hoist  elements  as  the  ammunition  is 
loaded  or  off-loaded  from  the  storage  area. 

Schematic-Based  Simulation 
The  Hoist  Trainer  simulation  was  derived  from 
hand-colored  schematic  drawings.  FMC 
developed  a  set  of  graphical  icons  and  arranged 
them  on  the  computer  screen  to  resemble  the 
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original  schematic  drawing.  These  icons  were 
developed  using  a  modified  Steamer 
environment  [7]. 

Color  was  incorporated  to  distinguish  different 
hydraulic  pressures.  The  color  scheme  selected 
was  representative  of  the  scheme  used  by  field 
service  technicians.  Red  was  used  to  depict 
high  hydraulic  pressure  and  yellow  was 
selected  to  represent  low  hydraulic  pressure. 
The  icons  were  linked  to  the  simulation  lists 
produced  by  the  counterfactual  inference  engine 
and  the  qualitative  model  to  produce  a 
simulation  that  animates,  step  by  step,  the 
mechanical  and  hydraulic  operation  of  the 
lower  hoist  assembly. 

Lessons  Learned 

When  FMC  originally  developed  the  Hoist 
Trainer,  a  system  developer  had  to  handcraft  all 
the  graphical  elements  of  the  lower  hoist. 
Each  graphical  element  of  the  simulation 
contains  two  parts:  a  view  describing  how  the 
component  is  displayed  on  the  screen  and  the 
design  or  structural  knowledge  that  describes 
its  physical  states.  Writing  this  code  for  every 
element  of  a  hydraulic/mechanical  system  can 
be  a  very  time  consuming  and  tedious  task!  In 
order  to  develop  future  systems  comparable  to 
the  Hoist  Trainer  in  a  cost-effective  manner, 
toe’:  must  be  developed  to  automate  the 
creation  of  qualitative  models  and  the  display 
and  operation  of  graphical  icons. 

Software  Tools 

FMC's  objective  with  respect  to  software  tools 
was  to  allow  system  developers  to  concentrate 
their  efforts  on  constructing  the  appropriate 
solution  for  a  problem  rather  than  being 
burdened  with  the  "mechanics"  of  developing 
the  underlying  software.  To  accomplish  this 
objective,  software  tools  were  developed  to 
help  build  future  applications  cost  effectively. 
To  aid  in  the  development  of  future  training 
systems,  FMC  began  the  development  of  two 
editors:  the  Icon  Editor  and  the  Connection 
Editor. 

Icon  Editor 

The  Icon  Editor  was  developed  to  allow  a 
developer  to  systematically  build  the  graphical 
elements  needed  to  simulate  the  operation  of  a 
mechanical  system.  Currently,  the  Icon  Editor 
is  a  generic  drawing  editor  that  allows  for  the 


selection  of  the  basic  geometric  objects.  Icon 
Editor  functions  were  developed  to  manipulate 
the  objects  and  ease  icon  development  and 
placement  on  the  computer  screen.  The  result 
is  a  generic  drawing  editor  suitable  for  creating 
graphical  icons  for  model-based  or  simulation 
training  applications.  Source  code  generators 
design^  for  the  Icon  Editor  will  automatically 
produce  LISP  code  for  the  graphical 
representation  of  an  icon  or  schematic-based 
element,  as  well  as  produce  code  for  the 
qualitative  model  representation. 

Connection  Editor 

The  Connection  Editor  complements  the  Icon 
Editor.  The  Connection  Editor  allows  for  the 
interactive  specification  of  the  hydraulic 
connections  drawn  by  the  Icon  Editor.  Using 
the  Connection  Editor,  a  system  developer  can 
methodically  create,  with  a  mouse  and  pull 
down  menus,  the  hydraulic  connections  for  the 
graphical  simulation.  An  integrated  source 
code  generator  creates  LISP  code  for  the 
graphical  display  thus  freeing  the  developer 
from  this  tedious  task.  The  result  is  a  generic 
software  tool  that  reduces  development^  costs 
for  future  training  systems. 

OfhfrApplicatiQns 

The  technical  concepts  represented  by  the  Hoist 
Trainer  are  generic  and  applicable  to  any 
mechanical/hydraulic  system.  For  example, 
within  the  FMC  product  line,  training 
applications  could  be  developed  for  the  Mark 
13  missile  launcher  or  for  all  assemblies  of  the 
Mark  45  naval  gun. 

The  architectural  components  of  the  Hoist 
Trainer  can  extend  to  other  application  domains 
such  as  model-based  diagnostics  [5]  or 
Intelligent  Tutor  Systems  (ITS).  The  Hoist 
diagnostic  application  uses  the  qualitative 
model,  the  counterfactual  engine,  and  the 
schematic-based  simulation.  The  qualitative 
model  works  with  the  counterfactual  engine  to 
create  a  list  of  possible  suspects.  The  suspect 
list  is  linked  to  the  schematic-based  simulation 
to  give  the  user  a  visual  depiction  of  the 
diagnostic  process  highlighting  the  suspect 
elements. 

The  qualitative  model  and  the  user  interface 
also  contribute  to  the  development  of  a  lower 
hoist  intelligent  tutor  system.  The  qualitative 


264 


model  makes  up  the  domain  knowledge  of  the 
ITS  and  the  schematic-based  simulation  is  used 
to  communicate  lower  hoist  operation. 

Outside  of  FMC,  applicable  application  areas 
include  aircraft,  automobile,  and  submarine 
systems.  The  qualitative  models  developed  for 
these  applications  could  use  the  counteifactual 
inference  engine,  since  it  is  a  general-putpose 
inference  engine  suitable  for  solving  many 
types  of  problems.  The  technical  concepts 
represented  by  the  qualitative  model  of  the 
Hoist  Trainer  are  also  generic.  The  qualitative 
representation  of  a  piston  in  one  training 
system  will  require  a  similar  level  of  abstraction 
in  another  application.  The  user  interface  is 
applicable  to  other  intelligent  training  systems 
because  the  simulation  of  normal  operation  is  a 
natural  way  to  communicate  the  underlying 
technology  to  the  student. 

Conclusion 

The  Hoist  Trainer  is  an  intelligent  computer- 
based  training  system  designed  to  help  students 
learn  the  internal  operation  of  the  Mark  45 
naval  gun.  The  concepts  implemented  in  the 
training  application  are  generic  and  applicable 
to  any  mechanical  system.  FMC  software 
tools,  with  designed  source  code  generators, 
ease  the  development  of  future  intelligent 
training  applications  and  offer  a  comprehensive 
software  engineering  approach  for  the 
development  of  intelligent  training  systems. 
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TAILORING  EXPLANATION  TO  USERS 


Anat  Jacoby 

To  understand  how  to  tailor  expert  system  explanation  to 
users,  a  study  was  conducted  that  examined  the  effects  of  goals  and 
explanations  on  different  task  performance.  Fifty-four  high  school 
students  were  randomly  assigned  to  one  of  two  explanation  groups  or 
a  control  group.  All  groups  received  the  same  purpose/goal  of 
needing  to  replicate  a  scheduling  task  of  assigning  flights  to 
gates  in  an  airport.  All  groups  received  a  sample  schedule,  and 
the  explanation  groups  received  in  addition  one  of  two 
explanations,  a  retrospective-trace  explanation  or  a 
reconstructive- justification  explanation.  The  retrospective-trace 
explanation  provided  the  detailed  steps  of  how  to  construct  a 
schedule;  the  reconstructive- justification  explanation  provided 
justification  of  a  given  schedule.  All  groups  received  two  tasks, 
in  random  order:  a  replication  task  of  producing  a  schedule  and 
a  verification  task  of  verifying  the  correctness  of  a  schedule. 
In  addition,  a  questionnaire  was  given  to  determine  the  students' 
satisfaction  from  the  explanations.  As  predicted,  students  getting 
the  retrospective-trace  explanation  performed  better  on  the 
replication  task  than  the  two  other  groups.  These  preliminary 
findings  suggest  that  explanation  should  be  tailored  to  people's 
goals  that  reflect  the  tasks  they  need  to  achieve. 
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TAILORING  EXPLANATION  TO  USERS 


Anat  Jacoby 


Expert  systems  are 
becoming  more  and  more  popular 
In  various  areas,  both  as  on 
the  job  tools  and  for 
education.  One  feature  thought 
to  be  central  to  the  success  of 
expert  systems  is  their  ability 
to  offer  explanations  of  their 
knowledge  and  reasoning. 
Explanation  is  essential  for 
understanding,  debugging, 
education,  acceptance,  and 
persuasion.  The  artificial 
intelligence  (AI)  /expert  system 
community  has  been  researching 
techniques  to  build  explanation 
facilities  for  expert  systems. 
Several  techniques  have  been 
developed  and  explanations  in 
the  form  of  software  have  been 
built.  Many  researchers  agree 
that  it  is  important  to  tailor 
the  explanation  to  the  user, 
but  the  approaches  taken  to 
implement  this  purpose  vary. 

It  seems  likely  that 
whether  an  explanation  will  be 
useful  or  not  will  depend  upon 
factors  such  as  the  type  of 
explanation  provided,  the 
nature  of  the  task,  and  the 
ability  and  goals  of  the  user 
(Berry  &  Broadbent,  1987) . 
Most  existing  reports,  however, 
consist  of  informal  information 
from  users.  Researchers  usually 
develop  a  technique  and  build 
the  explanation  facility 
without  studying  it  with 
people.  This  study  was 
conducted  to  investigate  the 
results  of  using  two  different 
explanation  approaches  on 
different  users.  The  main 
objective  was  to  try  to  "fit” 
the  explanation  to  the  users  by 
trying  different  explanations 
with  users  having  different 


goals  before  implementing  the 
explanation  facility. 

The  first  part  of  this 
paper  will  discuss  different 
explanation  techniques 
developed  by  artificial 
intelligence/  expert  system 
researchers,  and  different 
characteristics  of  expert 
system  users.  The  study  and  the 
hypotheses  will  be  introduced 
followed  by  methods  and 
results  sections  where  detailed 
discussion  will  be  provided. 

Explanation  Facility  Techniques 
The  most  common  technique  used 
for  developing  an  explanation 
facility  is  what  Waterman 
(1986)  describes  as 
retrospective  reasoning, 
mxetrospective  reasoning 
involves  tracing  through  the 
rules  ofmxhe  expert  system  to 
provide  the  user  the  line  of 
reasoning  that  the  expert 
system  used  in  its  solution. 
The  chain  or  sequence  of  rules 
that  led  to  the  conclusion  is 
provided.  Early  systems  like 
MYCIN  (Buchanan  &  Shortliffe, 

1984)  give  the  information 
stated  in  the  expert  system 
rules  translated  to  English. 
More  advanced  systems  attempt 
to  adapt  the  explanation  to  the 
user  by  providing  an 
interactive  dialogue  with  the 
user  (Moore  &  Swartout,  1989) , 
tailoring  the  explanation  to 
the  user's  "point  of  view" 
(McKeown,  Wish,  &  Matthews, 

1985)  ,  or  phrasing  the  text  to 
the  user's  level  of  expertise 
(Bateman  &  Paris,  1989) . 
Although  the  above  systems  vary 
in  their  emphasis,  all  share 
the  general  approach  to 
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explanation  of  following  the 
system's  line  of  reasoning. 
They  all  provide  a  trace  of  the 
steps  the  system  followed  to 
arrive  at  the  solution. 

A  different  approach  to 
explanation  development  was 
taken  by  Wick  and  Thompson 
(1989,  1990).  They  believe 
that  the  "line  of  explanation" 
should  be  different  from  the 
"line  of  reasoning".  a 
technique  was  proposed  named 
"reconstructive  explanation" 
which  uses  knowledge  other  than 
the  trace  as  the  basis  for  the 
explanation.  The  explanation 
does  not  give  the  details  of 
how  the  system  arrived  at  its 
conclusion  but  rather  justifies 
the  solution  by  giving  reasons 
why  it  is  the  correct  one.  For 
example,  in  explaining  the 
cause  of  an  excessive  load  on  a 
concrete  dam,  rather  than 
following  the  reasoning  path 
that  led  to  the  solution  of 
erosion,  a  reconstructive 
explanation  moves  directly  to 
erosion  and  explains  the 
relationships  that  bond  the 
symptoms  directly  to  erosion, 
and  introduces  new  data  that 
was  not  included  in  the 
solution  trace.  This  type  of 
explanation,  according  to  the 
authors,  is  more  appropriate 
for  many  users  of  expert 
systems.  Again,  this  technique 
has  been  implemented  in  a 
prototype  expert  system,  but 
its  effects  on  performance  have 
not  been  tested. 

Expert  System  Users 
Researchers  believe  that  there 
are  several  overlapping  reasons 
for  wanting  an  expert  system  to 
explain  its  reasoning.  These 
are:  understanding,  debugging, 
education,  acceptance,  an'^ 
persuasion.  These  rei  sons  were 
suggested  by  Buchanan  and 
Shortliffe  (1984)  and  are  cited 


by  many  other  explanation 
facility  researchers.  These 
points  will  be  further  explored 
in  an  attempt  to  understand  who 
are  the  typical  users  of  an 
expert  system  and  its 
explanation. 

Understanding  implies 
understanding  the  content  of 
the  knowledge  base  and  of  the 
line  of  reasoning. 
Understanding  is  important  for 
both  maintaining  and  using  the 
system.  Explanation  is  needed 
for  debugging,  as  expert 
systems  are  usually  built 
incrementally.  Debugging  is 
done  by  expert  system  builders 
and  maintainers.  Education  is 
needed  for  nonexperts  using  the 
system  since  users  who  feel 
they  learn  something  by  using 
the  system  are  likely  to  uae  it 
again.  Acceptance  refers  to 
convincing  potential  users  to 
use  the  system,  and  persuasion 
refers  to  convincing  users  that 
the  system's  conclusion  is 
correct.  Acceptance  and 
persuasion  are  closely  linked 
and  apply  to  people  who  are 
potential  or  actual  users  of 
the  system. 

Two  types  of  users  are 
reflected  from  the  above 
description;  they  are  usually 
referred  to  as  "knowledge 
engineers"  and  "end  users".  A 
knowledge  engineer  is  one  who 
builds  the  system  and  maintains 
it.  An  end  user  is  the  person 
using  the  system.  The 
knowledge  engineer  needs  the 
detailed  information  of  how  the 
system  arrived  at  its 
conclusion  in  order  to  debug 
and  expand  the  system.  The  end 
user  typically  uses  the  expert 
system  as  a  wJUFultant  program 
and  needs  to  understand  its 
reasoning  well  enough  to  be 
able  to  accept  responsibility 
for  the  solution.  Both  users 
need  the  explanation  facility 
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but  their  goals  for  nr  ading  it 
are  very  different.  .'hile  the 
knowledge  engineer  needs  to 
understand  the  process  of 
replicating  the  solution,  the 
end  user  is  interested  in 
verifying  the  solution  only. 
The  end  user  does  not  need  the 
information  of  how  exactly  the 
system  arrived  at  its 
conclusion.  Moreover,  this 
information  might  confuse  and 
overwhelm  the  end  user.  The 
end  user  only  needs  and  wants 
information  that  will  persuade 
him  or  her  that  the  solution  is 
correct.  The  knowledge 
engineer,  on  the  other  hand, 
needs  all  the  steps  of  how  the 
system  reasoned,  starting  at 
the  very  beginning  and 
following  every  trial  to  the 
solution. 

The  proposed  study 
suggests  two  different  lines  of 
explanation  to  these  two 
di*-*erent  users  having 
difrux.ent  needs  and  goals. 

The  Study  Two  different  kinds 
of  typical  users  were 
identified:  knowledge  engineers 
and  end  users.  These  two  user 
types  were  also  identified  as 
having  different  goals; 
replication  vs.  verification. 
The  knowledge  engineer  is 
interested  in  replication 
knowledge  while  the  end  user  is 
■  iterested  in  verification 
knowledge.  Two  kinds  of 
explanation  techniques  were 
introduced;  retrospective  and 
reconstructiv»  The  goal  of 
this  study  was  to  check  whether 
the  two  kinds  of  explanation 
natch  the  two  kinds  of  users, 
lu  was  believed  that  since  the 
retrospective  explanation 
provides  a  detailed  sequence  of 
the  steps  taken  towards  finding 
a  solution,  this  kind  of 
expl=»nation  will  be  appropriate 
for  rep’ icaticn.  The 


reconstructive  explanation,  on 
the  other  hand,  gives  a 
justification  for  the  solution, 
without  going  into  the  details 
of  the  process  used  to  arrive 
at  the  solution.  This 
explanation  was  believed  to  be 
more  appropriate  for 
verification  purposes. 

This  study  checked  the 
influence  of  the  replication 
goal  and  the  different 
explanations  on  user's 
satisfaction  and  perfoirmance. 
Another  study  needs  to  be  done 
to  compare  the  differences  in 
satisfaction  and  performance 
between  verification  goal  and 
replication  goal. 

The  purpose  instruction 
giv'en  to  subjects  in  this  study 
was  believed  to  direct  their 
attention  to  information 
relevant  to  the  task  to  be  done 
at  the  end  of  the  treatment. 
The  different  tasks  were 
replication  and  verification. 
The  replication  task  asked 
subjects  to  solve  a  problem  by 
carrying  out  a  procedure.  The 
verification  task  asked 
subjects  to  verify  the 
correctness  of  a  solution  to  a 
given  problem.  Two  kinds  of 
explanation  were  given  as 
treatments.  The  first,  a 
retrospective  trace,  stated  a 
general  problem  and  showed  the 
procedure  for  solving  it 
including  false  trials.  The 
second,  reconstructive 
justification .  stated  a 
solution  to  a  problem  with  a 
justification  but  did  not 
specify  all  the  details  of  the 
procedural  process  needed  to 
actually  solve  the  problem.  A 
control  group  received  no 
strategic  explanation  only  the 
information  needed  to  solve  the 
rasks.  The  domain  chosen  for 
the  study  was  that  of  assigning 
airplanes  to  airport  gates 
after  they  have  landed. 
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Hypotheses  It  was  hypothesized 
that  the  group  getting  the 
replication  purpose  with  the 
retrospective  trace  explanation 
would  perform  better  on  the 
replication  task  than  all 
other  groups,  since  the  goal, 
the  explanation,  and  the  task 
"match".  It  was  also 
hypothesized  that  students 
receiving  the  replication 
purpose  would  be  more  satisfied 
with  the  retrospective  trace 
explanation. 

Finally,  the  study  was 
conducted  because  its  results 
may  have  implication  for 
developing  good  explanation 
facilities  for  expert  systems 
as  well  as  for  teaching  and 
explaining.  Implications  for 
teaching  and  explaining  may  be 
for  deciding  of  whether  to 
provide  purpose/goal 
statements,  and  for  gearing  an 
explanation  to  one's  purpose/ 
goal.  For  the  area  of  expert 
systems  and  explanation 
facilities  although  the  study 
was  a  pencil  and  paper  study 
rather  than  an  actual  working 
expert  system,  it  can  provide 
theoretical  framework  for 
building  an  explanation 
facility. 

Pilot  Test  A  small  pilot  test 
was  conducted  using  as  subjects 
a  graduate  and  three  good  high 
school  students  attending  a 
private  high  school.  The 
purpose  of  the  pilot  test  was 
to  check  the  measures  and 
materials  as  well  as  the  time 
needed  for  task  completion.  The 
graduate  student  was  pilot 
tested  first  and  then  materials 
and  measures  were  changed  since 
the  decision  was  made  to 
conduct  the  study  in  a  high 
school  with  time  limit  of  one 
class  session.  The  pilot 
study's  results  showed  that  the 


fifty  minutes  available  would 
be  sufficient  and  the 
instructions  were  clear. 
Subjects  only  looked  at  the 
explanation  briefly  and  then 
proceeded  to  do  the  task,  going 
back  to  the  explanation  as 
needed.  Therefore,  the 
students  in  the  study  were 
given  specific  time  to  read  the 
example  and  explanation  before 
doing  the  task.  All  high 
school  pilot  subjects  did  the 
replication  task  with  no 
mistakes,  and  two  pilot 
subjects  had  one  mistake  in  the 
verification  task,  the  third 
had  no  mistakes. 


Subjects  Fifty  eight  male  and 
female  students  from  a  Los 
Angeles  public  high  school 
participated  in  the  study. 
Four  of  the  students  who  were 
not  proficient  in  English  were 
excluded  from  the  study 
analysis.  The  students  were 
tenth,  eleventh  and  twelfth 
graders  that  were  enrolled  in 
one  of  the  three  classes:  basic 
math,  geometry,  or  year  book. 
The  study  was  completed  during 
regular  class  time.  The 
students  were  informed  that 
they  were  participating  in  a 
study  and  to  increase  their 
motivation  to  participate  and 
do  well  their  teacher  told  them 
they  would  get  credit  for 
participating.  All  students 
received  the  replication 
purpose  and  were  randomly 
assigned  to  one  of  the 
explanation  groups  or  the 
control  group. 

Experimental  Design  The  design 
of  the  study  consisted  of  two 
explanation  groups  and  a 
control  group.  There  were  two 
dependent  measures  which  were 
given  after  the  treatments  and 
were  given  in  a  random  order  so 
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all  together  there  were  six 
different  groups:  Retrospective 
trace  explanation 
verification  task  first  (TV)  , 
retrospective  trace  explanation 

-  replication  task  first  (TR) , 
reconstructive  justification 
explanation  -  verification  task 
first  (RV) ,  reconstructive 
justification  explanation 
replication  task  first  (RR)  , 
control  group  -  verification 
task  first  (CV) ,  control  group 

-  replication  task  first  (CR) . 
Students  were  randomly  assigned 
to  treatments  within 
classrooms . 

All  subjects  received  the 
same  purpose/goal  instruction 
of  replication;  i.e.  that  they 
will  be  getting  an  example  and 
explanation  of  how  to  schedule 
airplanes  to  gates  and  would  be 
required  to  do  a  similar  task. 

The  explanations  given  to 
subjects  were:  retrospective 
trace,  reconstructive 
justification,  or  no 
explanation.  The 

retrospective  trace  explanation 
included  a  problem  and  the 
procedure  to  solve  the  problem; 
starting  from  the  problem  and 
working  towards  the  solution. 
Different  hypotheses  and 
attempts  for  solutions  were 
given  even  if  they  did  not  lead 
to  the  correct  solution.  The 
reconstructive  justification 
explanation  included  a  problem, 
its  solution  and  an  explanation 
which  justified  why  the  chosen 
solution  was  the  correct  one 
given  the  information  and 
constraints  or  rules.  False 
attempts  to  the  solution  were 
not  mentioned. 


Explanation 


Retrospective-  Reconstructive-  Control 

trace  justification 


Replicate 

Purpose 

Verify  first 
TV 

— 

Verify  first 
RV 

Verify  first 
CV 

Instruction 

Replicate 

Replicate 

Replicate 

first 

first 

first 

TR 

RR 

CR 

Procedure  The  subjects  were 
told  by  the  experimenter  that 
they  would  solve  a  problem  as 
part  of  a  research  study  and 
that  they  would  get  credit  for 
participation.  The 
experimenter  explained  the 
nature  of  the  task  of  assigning 
airline  flights  to  gates  after 
they  have  landed.  The  students 
were  told  that  they  would  be 
getting  different  tasks  and 
therefore  to  work  individually. 
Subjects  then  received  their 
material  and  were  given  five 
minutes  to  look  at  the  first 
packet  which  included  an 
example  and  the  appropriate 
explanation.  This  was  done  to 
insure  that  the  students  read 
the  explanation  before 
attempting  to  complete  the 
tasks  themselves.  Then, 
students  were  asked  to  solve 
the  two  tasks  and  fill  out  the 
steps  taken  to  solve  the  tasks 
and  the  questionnaire.  The 
experimenter  checked  that  all 
parts  were  completed  when  the 
students  handed  in  their  forms. 
The  study  took  one  class  period 
of  fifty  minutes  all  together, 
including  the  experimenter's 
explanation. 

Materials  All  subjects 
received  two  packets  of 
material.  The  first  packet 


included  the  flight  scheduling 
restriction,  an  example,  and 
for  the  explanation  groups-  the 
appropriate  explanation.  The 
second  packet  included  the 
replication  task,  the 
verification  task,  and  a 
questionnaire.  Following  each 
task  a  worksheet  was  given  and 
a  sheet  to  report  the  steps 
taken  to  solve  the  problem. 
The  packets  were  clipped 
together  so  that  the  second 
packet  was  up-side  down.  This 
was  done  since  the  students 
were  asked  to  read  the 
explanation  first  and  only 
start  working  on  the  tasks  when 
instructed.  Further 
descriptions  of  the  materials 
are  provided  below. 

The  domain  chosen  for  the 
study  was  the  scheduling  of 
planes  to  gates  in  an  airport. 
Airlines  have  specific  gates  in 
airports  that  they  can  use  for 
their  flights.  The  airline  is 
responsible  for  scheduling 
incoming  and  outgoing  flights 
to  the  gates .  There  are 
certain  rules  and  constraints 
to  follow  while  doing  this 
task.  One  constraint  is  that 
there  are  different  sizes  of 
aircrafts  and  different  sizes 
of  gates  and  the  sizes  need  to 
match,  i.e.,  one  cannot  assign 
a  big  aircraft  to  a  small  gate. 
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Another  constraint  is  that 
incoming  international  flights 
need  to  be  assigned  to  a  gate 
that  is  in  the  customs  area 
since  passengers  need  to  go 
through  passport  check  and 
customs  on  their  way  into  the 
country . 

Ground  controllers  are 
responsible  for  this  gate 
assignment  task.  An  expert 
system  was  developed  to  help 
ground  controllers  in  this 
task.  The  expert  system  was 
developed  for  TWA,  for  JFK  and 
St.  Louis  airports  (Brazile  & 
Swigger,  1988,  1989).  The 
system  does  not  currently  have 
an  explanation  facility  in  it. 

This  task  was  chosen  since 
it  represents  a  problem  solved 
by  expert  systems  and  is  a  task 
that  can  be  replicated  (to  make 
a  schedule)  and  verified  (to 
check  the  schedule)  by  a 
subject  pool  without 
specialized  domain  knowledge. 
The  domain  is  simple  to 
understand  since  it  uses 
everyday  language  and 
terminology.  The  task, 
although  simple,  is  one  most 
people  haven't  attempted. 

All  students  received  the 
same  purpose  statement  and  the 
same  restriction  list  and 
sample  schedule.  The  purpose 
statement  given  to  all  subjects 
was  the  replication  purpose/ 
goal.  All  subjects  were  told 
they  would  be  receiving  an 
example  and  explanation  and 
would  later  have  to  complete  a 
similar  task  themselves.  The 
objective  of  the  purpose 
statement  was  to  manipulate 
subjects'  purpose/goal  of  why 
they  were  receiving  the 
information  and  explanation 
that  they  were.  Another  study 
should  be  done  to  check  the 
verification  purpose. 

The  standard  materials 


included  a  page  of  flight 
scheduling  restrictions,  an 
example,  and  for  the 
explanation  groups,  the 
appropriate  explanation.  The 
flight  scheduling  restrictions 
included  gate  restriction, 
i.e.,  domestic  and 
international  gates,  and 
separation  time  between  two 
different  flights  using  the 
same  gate;  plane  type 
restriction,  i.e.,  five 
different  plane  types  and  what 
domestic  and  international 
gates  they  could  be  assigned 
to;  and  taxiway  access 
restriction,  i.e.,  what 
different  taxiway  exists  and 
what  gates  they  lead  to.  The 
example  consisted  of  ten 
incoming  flight  information  and 
the  gate  the  flights  were 
assigned  to.  The  format  of  the 
example  was  identical  to  the 
format  of  the  tasks.  The 
explanations  were  given  to  the 
appropriate  treatment  group  and 
provided  an  explanation  of  the 
example.  The  control  group 
received  no  explanation. 

Dependent  Measures  Subjects 
completed  two  tasks:  a 
replication  task  and  a 
verification  task,  and  a 
questionnaire.  The 
replication  task  provided  a 
list  of  flight  information  that 
included  the  following  facts: 
whether  the  flight  was  domestic 
or  international,  plane  type, 
flight  number,  arrival  time, 
and  departure  time.  An  empty 
column  was  given  for  the 
subjects  to  fill  in  the  gate 
assignment.  Subjects  were 
asked  to  schedule  ten  flights 
to  the  appropriate  gates 
without  violating  any  of  the 
constraints.  A  worksheet  was 
provided,  and  a  sheet  asking 
subjects  to  report  the  steps 
they  took  to  solve  the  task. 
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The  verification  task  provided 
a  similar  list  of  flight 
information  and  also  included 
the  same  facts  of  whether  the 
flight  was  domestic  or 
international,  plane  type, 
flight  number,  arrival  time, 
departure  time,  and  the  gate 
assignment.  For  this  task  the 
gate  assignment  was  given  and 
subjects  were  asked  to  verify 
the  schedule  for  ten  flights 
and  circle  flights  that 
violated  restrictions.  Of  the 
ten  scheduled  flights  given, 
three  had  the  wrong  assignment 
that  were  due  to  violating 
different  restrictions.  There 
was  a  worksheet  and  a  page  to 
write  down  the  steps  taken  to 
solve  the  problem  similarly  to 
the  replication  task.  The  tasks 
were  paper  and  pencil  tasks. 

The  questionnaire  asked 
all  subjects  whether  the 
explanation  they  received  was 
helpful  in  doing  the 
replication  task  or  the 
verification  task,  and  why. 
The  questionnaire  also  asked  if 
the  example  and  explanation 
were  helpful  in  achieving  the 
stated  goal.  General  questions 
were  also  asked  about  sex,  year 
in  school,  GPA,  and  domestic, 
and  international  travel 
experience. 

Treatment  Description  There 
were  two  kinds  of  explanations; 
retrospective  trace  and 
reconstructive  justification. 
The  retrospective  trace 
explanation  explained  to 
subjects  how  to  make  a  schedule 
by  starting  at  the  information 
given  of  flights,  gates,  and 
constraints,  and  trying  to 
assign  flights  to  appropriate 
gates.  The  explanation 
included  strategies  such  as 
assign  all  big  planes  first, 
and  leave  gates  that  are  close 
to  customs  for  incoming 


international  flights  etc.  The 
explanation  demonstrated  how 
the  example  schedule  was  put 
together  showing  false  trials 
as  well.  That  is,  a  flight 
might  be  assigned  a  gate  and 
then  needs  to  be  reassigned 
because  of  a  different  flight 
coming  in  with  more 
restrictions.  A  graphic  table 
was  shown  and  explained.  The 
table  had  a  horizontal  column 
of  the  available  gates  with  the 
matching  taxiways,  and  a 
vertical  column  of  the  times. 
Lines  were  drawn  to  show  the 
gate  was  occupied  at  a  given 
time.  The  table  provided  a 
visualization  strategy  of 
solving  the  task. 

The  reconstructive 
justification  explanation 
justified  the  example  schedule 
by  explaining  that  it  fits  all 
the  rules  and  constraints. 
There  was  not  a  specific 
explanation  of  the  process 
taken  to  arrive  at  the 
solution,  and  false  trails  were 
not  mentioned. 

RESULTS  The  verification  and 
replication  tasks  were  checked 
for  correctness.  The  recorded 
score  represents  the  number  of 
mistakes  made.  The  two  tasks 
were  very  different  in  nature 
and  therefore  no  comparison  can 
be  made  between  the  tasks,  only 
between  the  experimental 
groups',  among  each  task.  Table 
1  shows  the  means  and  standard 
deviations  of  both  tasks  for 
each  experimental  group. 


274 


Table  1 


Performance  on  the  verification  and  replication  task  of  all 
explanation  groups 


Explanation 

Verification 

task 

Replication 

task 

Retrospective-trace 
verification  first 

M 

(^) 

1.57 

(0.54) 

3.14 

(0.38) 

Retrospective-trace 
replication  first 

M 

(m) 

1.89 

(0.60) 

2.00 

(1.34) 

Reconstructive- justif ication 
verification  first  M  ( SD) 

1.57(0.54) 

3.43 

(0.79) 

Reconstructive-j us tif ication 
replication  first  M  (SD) 

1.89(0.99) 

3.40 

(1.27) 

Control 

verification  first 

M 

(SD) 

1.90 

(0.74) 

3.00 

(1.41) 

Control 

replication  first 

M 

(SD) 

1.57 

(0.79) 

4.44 

(0.52) 

Note:  The  scores  represent  the  number  of  mistakes  made. 


A  two  way  ANOVA  that 
checked  the  effects  of  the 
explanation  groups  and  the 
order  showed  a  significant 
explanation  (group)  main 
effect,  no  order  main  effect 
and  a  group  by  order 
interaction  for  the  replication 
task  (see  table  2)  .  No 

significant  main  effects  nor 
interactions  were  found  for  the 
verification  task. 
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Table  2 


AKOVA  table  to  show  explanation  group  main  effect 


interaction  for  the  replication  task 


*  orde 


Source 

df 

Sum  of 
Sauares 

Mean 

Souare 

Z 

Grp  ( exp . ) 

2 

15.72 

7.86 

6.60* 

Order 

1 

0.26 

0.26 

0.22 

Grp  *  Order 

2 

15.22 

7.61 

6.39* 

Error 

48 

57.19 

1.19 

Total 

53 

88.15 

1.66 

Further  analysis  showed 
that,  as  hypothesized,  there 
was  a  significant  difference 
between  the  groups  in  the 
replication  task.  A  post  hoc 
Tukey  comparison  showed  that 
the  significant  difference  was 
between  the  retrospective-trace 
explanation  group  and  the 
reconstructive- justification 
explanation  group,  and  between 
the  r etr ospect i ve-trace 
explanation  group  and  the  no 
explanation  control  group. 
Students  who  received  the 
retrospective- trace  explanation 
performed  better  (made  fewer 
mistakes)  on  the  replication 
task.  No  significant 
differences  occurred  between 
the 
reconstructive- just if icat ion 
explanation  group  and  the 
control  group.  These  results 
demonstrate  that  the 
retrospective-trace  explanation 
was  the  appropriate  explanation 
for  the  replication  task. 

Although  not  hypothesized, 
data  analysis  also  checked  for 
differences  among  the  groups  in 
the  verification  task.  No 
significant  differences  were 
found  among  the  groups,  despite 
the  different  explanations 
received. 

The  ANOVA  also  showed  a 
group  by  order  interaction. 
There  was  a  significant 


difference  for  the  replication 
task  in  the  retrospective-trace 
explanation  and  the  control 
group.  While  in  the 
retrospective  trace  explanation 
students  who  received  the 
replication  task  first  did 
better  than  students  receiving 
the  replication  task  second, 
the  opposite  occurred  in  the 
control  group.  For  the 
verification  task,  although  not 
significant,  the  results 
indicated  that  the  students 
receiving  the  reconstructive- 
justification  explanation 
performed  better  on  the 
verification  task  when 
receiving  it  first,  while 
students  in  the  control  group 
performed  oppositely. 

Three  classes  participated 
in  the  study.  A  two  way  ANOVA 
found  a  significant  difference 
among  classes  on  their 
performance  on  both  tasks  (see 
tables  3  and  4) .  There  was  no 
interaction  with  explanation, 
and  therefore  these  results  are 
assumed  to  occur  from  the 
different  abilities  of  the 
students  in  those  three 
classes. 


Table  3 


ANOVA  table  to  show  class  main  effect  for  replication  task 


Source 

Sum  of 
Sauares 

Mean 

Sauare 

F 

Grp  (exp.) 

2 

16.52 

8.26 

6.74* 

Class 

2 

12.65 

6.33 

5.16* 

Grp  *  Class 

4 

4.84 

1.21 

0.99 

Error 

45 

55.17 

1.23 

Total 

53 

88.15 

1.66 

Table  4 

ANOVA  table 

to  show 

class  main  effect 

for  verification  task 

Source 

df 

Sum  of 

Mean 

F 

Sauares 

Sauare 

Grp .  ( exp . ) 

2 

0.07 

0.03 

0.10 

Class 

2 

8.12 

4.06 

11.64* 

Grp  *  Class 

4 

1.26 

0.32 

0.91 

Error 

39 

13.61 

0.35 

Total 

47 

23.00 

0.49 

A  two  way  ANOVA  was  done 
to  check  for  sex  differences. 
There  were  no  significant  sex 
differences . 

Chi  square  tests  were 
conducted  to  check  the 
questionnaire  answers.  Most 
students  recorded  that  the 
explanation  they  received 
helped  them  to  do  both  tasks, 
and  matched  the  goal  they 
received.  There  were  no 
significant  differences  among 
the  explanation  groups  in  the 
way  they  answered  the 
questionnaire . 


DISCUSSION  This  study  examined 
the  effects  of  students'  goal 
and  the  kind  of  explanation 
received  on  their  performance 
on  a  replication  task  and  a 
verification  task.  The  goal  of 
the  study  was  to  better 


understand  how  expert  system 
explanation  can  be  tailored  to 
users.  As  predicted,  students 
who  received  the  replication 
goal  and  the  retrospective 
trace  explanation  performed  the 
replication  task  better  than 
the  group  getting  the 
reconstructive- justification 
explanation,  and  the  control 
group  receiving  no  explanation. 

To  understand  further  the 
effect  of  both  the  goal  and  the 
explanation  on  users  a  study 
needs  to  be  conducted  where  a 
verification  goal  is  given  to 
half  of  the  subjects.  It  is 
predicted  that  these  subjects 
getting  the  verification  goal 
and  the 
reconstructive- justification 
explanation  will  do  better  on 
the  verification  task  than  all 
other  subjects. 
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There  was  no  significant 
differences  in  the  performance 
on  the  verification  task  among 
the  explanation  groups  in  this 
study.  To  further  understand 
if  this  was  due  to  the 
explanation  or  to  the  goal 
being  a  replication  goal,  a 
study  should  be  conducted  with 
another  domain  and  another 
explanation.  It  is  possible 
that  the  verification  task 
might  have  jeen  too  easy  and 
subjects  could  perform  it 
regardless  of  the  kind  of 
explanation  they  received. 

Another  interesting 
finding  was  that  the  order  in 
which  the  tasks  were  given 
resulted  in  differences  in 
performance.  While  subjects 
who  received  the  task  that 
matched  their  explanation  first 
(replication  task  for 
retrospective-trace 
explanation,  and  verification 
task  for 
reconstructive- justification) , 
did  better  than  subjects 
receiving  it  second,  the 
control  group  did  better  on 
that  task  when  receiving  it 
second .  The  author ' s 
explanation  to  that  is  that 
when  the  explanation  matched 
the  task  it  was  more  helpful  to 
do  the  task  right  away  than  get 
another  task  in  the  middle. 
For  the  control  group,  though, 
who  received  no  explanation, 
the  practice  of  doing  a  task 
helped  on  the  second  task. 

Altogether,  the  results  do 
show  that  a  goal  and  different 
explanations  do  effect 
subjects'  performance  on 
replication  and  verification 
tasks.  Further  research  needs 
to  confirm  the  findings  and 
make  them  more  specific.  The 
results  are  applicable  to  both 
classroom  teaching  and  to 
writing  good  explanations  in 
expert  systems .  To  make  the 


results  more  applicable  to  the 
different  areas  further 
research  needs  to  be  more 
specific  and  in  real  life 
settings  and  situations.  That 
is,  for  applications  in 
teaching,  real  classroom 
situations  need  to  be  examined 
so  that  students  actually  hold 
the  goals  to  be  manipulated, 
and  need  to  do  the  actual 
tasks.  Similarly  in  the  expert 
system's  domain,  a  computer 
should  be  used  instead  of 
paper  and  pencil,  and  different 
expert  system's  users  rather 
than  high  school  students 
should  serve  as  subjects. 
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Development  of  the 

Media  Elimination  and  Design  Intelligent  Aid 
Dr.  William  T.  Melton 
Abstract 

The  Training  Development  and  Analysis  Directorate  of  the 
United  States  Army  Training  and  Doctrine  Command  (TRADOC)  has 
developed  an  expert  media  selection  system  and  design  aid  to 
assist  soldiers  assigned  to  develop  training  in  making  critical 
design  decisions. 

The  Media  Elimination  and  Design  Intelligent  Aid  (MEDIA), 
which  will  be  demonstrated  assists  training  developers  by 
recommending  media  and  other  design  elements  to  be  used  in 
teaching  specific  tasks  or  learning  objectives.  Media, 

methods,  site,  learning  strategy,  categories  of  learning,  and 
learning  activities  are  selected  based  on  requirements  related 
to  the  task  or  objective.  The  model  lists  all  of  the  media 
which  provide  acceptable  levels  of  fidelity  and  interaction. 
The  acceptable  media  are  compared  and  prioritized  on  three 
ordinal  scales  reflecting  procurement,  development  and 
implementation  costs. 

This  paper  presents  the  general  logic  that  supports  the 
programming  of  these  design  decisions  based  on  a  limited  group 
of  questions  which  are  primarily  related  to  the  nature  of  the 
task  or  objective  being  learned  and  the  general  characteristics 
of  the  soldiers  to  be  trained.  Several  of  the  decision 
relationships  differ  significantly  from  those  presented  in  the 
traditional  instructional  systems  development  model. 
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DEVELOPMENT  OF  THE 
MEDIA  ELIMINATION  AND 
DESIGN  INTELLIGENT  AID 


Dr.  William  T.  Melton 


Background 

The  establishment  of  a 
systematic  process  of  making 
critical  design  decisions 
concerning  media,  methods, 
learning  strategy,  site 
selection,  categories  of 
learning,  and  events  and 
activities  has  for  many 
years  been  a  major  concern 
for  military  training 

developers.  The  Army  has 

purchased  and  used  a  variety 

of  paper-based  systems  and 
job  aids  to  help  developers 
to  make  these  decisions. 
These  have  been  used  with 
varying  degrees  of  success. 

The  development  of  an 
automated  system  to  provide 
assistance  in  making  these 
decisions  has  been 

considered  and  attempted 
several  times.  Some  of  the 
early  systems,  either 

provided  matrix  type 

templates  for  the  developer 
to  use  in  creating  his  own 
decision  making  system  or 
were  oriented  more  toward 
audit  trail  (decision 

rationale)  data  storage  than 
decision  making  assistance. 

The  advent  of  expert  and 
artificially  intelligent 

systems  has  encouraged  the 
consideration  of  other 
alternatives  for  providing 
decision  making  assistance 
to  the  training  developer. 


Several  expert  systems  for 
media  selection  have  been 
developed  in  recent  years. 
Those  surveyed  tended  to 
have  very  limited  media 
arrays  or  expressed  possible 
proprietary  bias.  This 

paper  will  present  the 

general  logic  supporting  the 
decision  making  aids 

incorporated  in  an  expert 
system  developed  by  the 

Training  Development  and 
Analysis  Directorate  of  the 
Army  Training  and  Doctrine 
Command  (TRADOC) 

specifically  designed  for 

military  use. 


Description  of  the  Media 

Elimination _ and _ Design 

Intelligent  Aid  TMEDIA) 


MEDIA  is  a  computer  expert 
system  designed  to  assist 


developers  in 
media,  methods, 
strategy,  site 
categories  of 
and  events  and 
Recommendations 
in  each  of  these 
areas  which  will  meet  all  of 
the  training  design 

requirements  of  the  task  or 
objective  to  be  trained. 


training 

selecting 

learning 

selection, 

learning 

activities. 

are  made 


Developers  using  the  program 
are  asked  a  series  of 
questions  about  the  task  or 
objective  for  which  they  are 
designing  training.  MEDIA 
uses  the  answers  to  these 
questions  to  eliminate  media 
and  methods  which  cannot 
provide  the  required  cues, 
practice,  interaction  or 
other  characteristics. 

MEDIA  also  uses  these  same 
questions  to  recommend 
training  sites,  learning 
strategy,  categories  of 
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learning  involved  and 
guidelines  for  appropriate 
events  and  activities. 


Madia _ and _ Method 

Recommendations 


Media  and  method 

recommendations  are  based  on 
a  series  of  questions 
primarily  about  the  nature  of 
the  task  or  objective.  The 
system  does  not  assume  that 
the  user  has  expertise  in 
training  design  or 

development.  To  enable  the 
system  to  be  used  effectively 
by  persons  who  are  not 

trained  in  education,  the 

questions  intentionally  avoid 
addressing  the  training 

requirements.  This  choice 
was  made  because  many  of  the 
persons  charged  with  making 
media  and  method  decisions 
are  technical  subject  matter 
experts  who  are  very  familiar 
with  the  task  but  are  not 
professional  trainers. 

The  questions  are  therefore 
structured  to  elicit 

information  specifically 

about  the  performance  of  the 
task  or  objective  related  to: 


1. 

the 

type  of  performer. 

2. 

the 

type  of  task. 

3. 

the 

interaction  required. 

4. 

the 

type  of  cues. 

5.  the  physical  and  mental 

skills  required. 

6. 

the 

use  of  job  aids,  and 

7. 

the 

task  characteristics. 

The  MEDIA  program  does  not 
try  to  select  a  single  best 
medium  or  method  for  each 
ta?;k  or  objective.  Rather  it 
provides  the  training 

developer  recommendations 

from  which  to  choose.  Based 
on  answers  to  the  questions, 
developers  are  provided 
cost- ordered  lists  of  media 
and  method  lists  for  each  of 
the  four  primary  levels  of 
training.  These  four  lists 
reflect  media  and  metnods 
appropriate  for  the 

introduction,  mental 

practice,  competency 

evaluation,  and  sustainment 
of  training  of  the  task  or 
objective. 


The  media  are  listed  in 
logical  cost  order  from  the 
least  to  the  most  expensive. 
Three  ordinal  scales  from  one 
to  seven  are  used  to  assign 
logical  comparative  costs  to 
each  medium  presented  for  its 
procurement,  development  and 
implementation  costs.  A 

development  time  value  of 
less  than  six  months,  more 
than  6  months  or  more  than  2 
years  is  also  given  for  each 
medium  listed.  The 

comparative  costs  and  time 
estimates  are  presented  to 
help  the  developer  to  make 
final  media  selections  from 
among  the  media  presented  in 
the  lists. 


Learning _ Strategy 

Recommendation 


The  program  recommends  one  of 
the  six  learning  strategies 
available  to  the  Army  based 
on  the  level  of  inter 
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activity  needed  to 

effectively  tra.’n  a  given 
task. 

Review  of  the  six  primary 
learning  strategies  available 
for  use  in  military  training 
revealed  that  ^ney  could  be 
related  to  each  other  by 
examining  the  change  in  the 
student  teacher  relationship. 
This  relationship  provided  a 
scale  of  changing  levels  of 
live  personal  human 

interaction. 

The  six  learning  strategies 
considered  listed  in  order  of 
increasing  human  interaction 
are  programmed  learning, 
traditional  learning, 

exercise  or  experimental 

learning,  small  group 

learning,  pure  group  process 
learning,  and  mentor 

learning.  Because  the  system 
user  is  often  not  a  trained 
educator,  descriptions  of 

each  of  these  are  presented 
to  the  user  as  they  are 
recommended. 

The  program  asks  the  training 
developer  three  primary 
questions  to  determine  the 
level  of  interaction  needed 
(thereby  determining  the 

particular  learning  strategy 
needed).  These  questions 

pertain  to  the  experience 
level  of  the  trainee,  the 
level  of  commitment  required 
of  the  graduate,  and  the 
anticipated  level  of 

resistance  to  the  training. 

Increasing  experience, 

required  commitment  and 

potential  resistance  to  the 


training  ail  resulted  in  an 
increased  need  for  live  human 
interaction.  The  cumulative 
effect  of  these  is  used  to 
recommend  learning  strategies 
with  an  appropriate  level  of 
live  human  interaction. 

The  learning  strategy  with 
the  least  acceptable  level  of 
live  human  interaction  is  the 
one  presented  to  the  user  as 
the  system  recommendation. 
The  presentation  of  the 
recommendation  is  followed  by 
a  statement  that  any  strategy 
with  a  higher  level  of  human 
interaction  is  also 

acceptable  but  will  probably 
be  more  resource  intensive. 

The  incorporation  of  a 
learning  strategy  model  is  a 
change  from  previous 

systematic  design  systems. 
It  is  included  because  the 
Army  has  experienced  several 
periods  of  change  in  which 
new  learning  strategies  were 
adopted  as  a  replacement  for 
a  previous  strategy  simply  to 
find  that  they  did  not  solve 
the  problem.  Each  learning 
strategy  has  an  appropriate 
use  with  a  particular  level 
of  student.  The  failure  of 
strategies  resulted  from 
attempting  to  use  them  for 
training  a  group  which  was 
not  appropriate  for  the 
strategy.  The  model  is 

designed  to  help  training 
developers  to  match  learning 
strategies  to  the  appropriate 
levels  of  student  need  for 
live  human  interaction. 
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site  Recommendation 

The  training  site 

recommendation  is  made  based 
on  the  elimination  of  less 
resource  intensive  sites. 
The  sites  are  considered  and 
selected  beginning  with  the 
least  expensive  location  and 
continuing  to  the  most 
expensive.  The  less 

expensive  location  is  only 
eliminated  if  the  answers  to 
the  questions  asked  by  the 
system  indicate  that  it  is 
not  usable. 

These  sites  are  eliminated 
through  rather  direct 

questions  on: 

1.  the  practicality  of  using 
job  aids, 

2.  the  ability  of  the  unit 

(job  site)  to  provide  the 
support  necessary  to  train 

the  task, 

3.  the  requirement  for 
proficiency  in  the  task  on 
initial  unit  (job  site) 
assignment,  and 

4.  the  necessity  for  an  on 

site  instructor  to  evaluate 
or  to  provide  for  safe 

training. 

Some  questions  on  the 

characteristics  of  the  task 

are  also  used  in  making  the 
site  selection 

recommendation . 

This  is  a  departure  from 
previous  site  selection 
systems  used  by  the  Army 
which  were  geared  toward 
identifying  reasons  the 


training  should  be  done  at 
the  school.  Recommending  the 
school  only  if  the  training 
cannot  be  done  elsewhere 
follows  a  logic  pattern 
borrowed  from  the  British 
Army  system  for  site 

selection  and  is  much  more 
realistic  in  times  of 

constrained  training 

resources. 

The  training  site 

recommendation  is  presented 
with  a  question  allowing  the 
user  to  choose  or  override 
the  recommendation.  This  is 
done  because  the  selection  of 
the  primary  training  site, 
the  site  where  competency 
evaluation  will  occur,  will 
effect  the  media  available 
for  use. 

The  sites  listed  for 
selection  include: 

1.  job  aid  in  the  unit  or 
living  area, 

2.  distributed  sites, 

3.  unit  (job  site) 
facilities, 

4.  school  facilities,  and 

5.  dedicated  permanent 
facilities. 

The  site  selected  will  appear 
as  the  site  for  the 
competency  evaluation  level 
of  training.  Other  sites  may 
be  recommended  for  the  other 
levels  of  training  if  they 
are  possible  and  less 
resource  intensive  than  the 
site  chosen  for  competency 
evaluation. 
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Categories  of  Learning  and 

Recommended _ Events _ and 

Activities  Guidelines 

The  questions  related  to  the 
mental  and  physical  skills 
required  for  the  task  or 

objective  list  verbs 

associated  with  the  three 
categories  and  eleven 

subcategories  of  learning. 

The  categories  of  learning 
are  physical,  mental  and 

attitudinal.  The  eleven 

subcategories  included  in  the 
system  are: 

1.  rule  learning, 

2.  classifying  patterns, 

3.  identifying  symbols, 

4.  detecting, 

5.  making  decisions, 

6.  recalling  bodies  of 

knowledge, 

7.  performing  gross  motor 
skills, 

8.  steering  and  guiding, 

9.  positioning  movement  and 
recalling  procedures, 

10.  voice  communicating,  and 

11.  attitude  learning. 

Those  subcategories  which  are 
related  to  the  verbs  the  user 
selects  as  characteristic  of 
the  task  or  objective  being 
evaluated  are  listed  on  a 
screen  at  the  end  of  the 
program. 


The  system  recommends  that 
the  user  use  the 

sub categories  of  learning 
listed  to  enter  the 

guidelines  and  flowcharts  in 
the  user's  manual  provided 
with  the  system.  The 

selection  of  the  appropriate 
guidelines  and  flowcharts 

will  provide  the  user  with  a 

tested  structure  of 

activities  useful  in 

providing  an  effective 

program  for  that  specific 
mental  or  physical  skill.  If 
many  subcategories  are  listed 
the  user  may  use  these  to 
create  logical  enabling 
objectives  with  which  to 

reenter  the  system  for  a  more 
refined  set  of  design 
recommendations. 


Summary 

The  primary  purposes  of  this 
design  aid  are  to  save 
training  developer  time,  to 
eliminate  costly  mistakes,  to 
support  the  effective 

distribution  of  training  and 
to  recommend  the  lowest  cost 
strategy,  media,  site  and 
method  which  can  be  used  to 
effectively  train  the  task  or 
objective. 

With  shrinking  resources  the 
military  training  developer 
will  be  pressed  to  produce 
more  training  products  in  a 
shorter  period  of  time. 
Training  developers  will  no 
longer  be  able  to  simply 
follow  the  latest 

technological  fad  or  "do  it 
the  way  we  always  did  it"  as 
a  quick  solution  to  design 
decision  making. 
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The  military  services  have 
already  experienced  an 
adequate  number  of  excursions 
into  media  which  did  not 
produce  the  results 

expected.  It  is  more 

important  than  ever  that 
training  developers 

systematically  tie  media  and 
design  decisions  to  the 
characteristics  of  the  task 
or  objective. 


The  cumbersome  paper  based 
systems  of  the  past  simply 
take  too  long  to  implement  to 
support  the  design  decision 
making  of  the  future.  The 
shear  number  of 

relationships,  available 

media  and  design  alternatives 
make  the  creation  of  new 
paper  based  design  decision 
systems  unworkable. 


The  design  decisions  in  MEDIA 
were  based  on  these  four 
principles. 


1.  The  ideal  strategy, 

method,  site,  and  media 
combination  is  an  expert 
performer/teacher  with  a 
single  student,  training  on 
the  actual  equipment  in  the 
job  performance  environment. 
All  training  designs  are 
attempts  to  simulate  the 
ideal  in  progressively 
constrained  environments. 

Media  are  therefore  at  best 
an  extension  of  this  kind  of 
live  human  interaction  to 


meet  the  needs  of  larger 
numbers  of  students  at  lower 
resource  expenditure. 

2.  Many  different  media, 
methods,  and  strategies  can 
effectively  train  parts  of 
almost  any  task.  Training 
experts  can  rarely  agree  on  a 
single  "best"  medium,  method 
or  strategy  for  training  any 
specific  task  or  objective. 
Most  media,  methods  and 
strategies  can  be  distributed 
to  multiple  locations. 

3.  The  major  criteria 

effecting  media  method  and 
strategy  selection  can  be 
grouped  under  the 

consideration  of  fidelity, 
interaction,  and  cost.  The 
lowest  cost  medium  and 
strategy  that  will  meet  the 
fidelity  and  interaction 
criteria  associated  with  the 
task  should  be  the  first 
recommended  solution.  The 
low  cost  design  solution 
should  be  recommended  to 
conserve  resources  to  support 
tasks  and  objectives 

requiring  greater  levels  of 
fidelity  and  interaction. 

4.  Any  training  media 

selection  for  the  military 
should  consider  the  needs  of 
the  active  and  reserve 
components,  mobilization, 

unit  training  and  sustainment 
of  proficiency. 
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INFORMATIONAL  FEEDBACK  AND  CONCEPT  LEARNING 


IN  COMPUTER-AIDED  INSTRUCTION 


Lieutenant  Colonel  Milton  Nielsen,  Ph.D. 
Chief,  Instructional  Technology 
Commandant  of  Cadets,  USAF  Academy,  CO 


Feedback  during  student  practice  is  considered  a 
fundamental  component  of  well  designed  Computer  Aided 
Instruction  (CAI).  This  study  investigated  the  effects 
of  three  levels  of  feedback  to  student  responses  to 
practice  questions  and  a  second  attempt  to  answer  the 
practice  question  in  a  CAI  lesson.  The  lesson  taught  a 
new  concept,  basic  navigation.  The  subjects  were  173 
college  freshmen  at  the  Academy.  Significant  results 
were  found  for  providing  the  learners  a  second  attempt 
to  answer  the  practice  questions  on  their  acquisition 
and  retention  tests  and  for  the  increased  levels  of 
feedback  to  error  in  answering  their  practice  questions 
on  their  retention  test. 

Lieutenant  Colonel  Nielsen  graduated  from  Texas  A 
&  M  University,  with  degrees  in  Political  Science. 

After  initial  aircrew  assignments  in  B-52's,  he  was 
assigned  to  the  USAF  Academy.  During  his  first  tour  at 
USAFA,  he  was  an  instructor  and  then  course  director 
for  the  senior  military  studies  course  as  well  as  the 
Chief  of  Evaluations  for  Military  Instruction.  He  was 
also  a  T-43  navigation  instructor  and  a  Cadet  Squadron 
Training  Officer.  He  returned  to  USAFA  and  his  current 
duties  after  a  doctoral  program  in  Instructional 
Technology  at  the  University  of  Texas,  Austin. 
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INFORMATIONAL  FEEDBACK  AND  CONCEPT  LEARNING 


IN  COMPUTER-AIDED  INSTRUCTION 

Introduction 

One  of  the  events  of  instruction  (R.  Gagne,  1974) 
which  poses  great  potential  for  a  robust  application 
within  CAI  is  feedback  (Brophy,  1986;  B.  Cohen,  1985; 
Kulhavy,  1977).  Generally,  feedback  in  CAI  is  an 
imbedded  software  reaction  to  the  learner's  responses 
to  questions  or  situations.  Feedback  could  take  the 
form  of  auditory,  graphic,  or  textual  messages.  Many 
aspects  of  feedback  have  been  researched,  however  many 
believe  it  to  be  one  of  the  weakest  areas  of 
instructional  software  (G.  Cohen,  1983;  J.  Merrill, 
1987;  Cohen,  1985;  Komoski,  1984).  In  fact,  the 
research  results  only  seem  to  confound  the  dialogue 
(Jonassen,  1988).  Part  of  the  problem  may  be  the 
diversity  of  the  approaches  taken  in  the  study  of 
feedback.  Feedback  has  been  examined  for  its  ability 
to  reinforce,  to  inform,  to  identify  errors,  and  to 
correct  errors.  It  has  been  researched  in  relationship 
to  other  factors  such  as  cognitive  level  of  the 
instructional  objective,  different  learner 
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characteristics,  and  level  of  learner  control.  A  brief 
examination  of  the  research  from  these  perspectives  is 
helpful . 

Early  views  of  feedback  came  from  the  operant 
conditioning  school  of  behavioralist  psychology. 
Feedback  was  reinforcement  used  to  strengthen  correct 
responses.  Positive  reinforcement  was  a  pleasant 
stimulus  given  immediately  after  a  correct  response. 
However,  later  research,  conducted  by  those  who  ascribe 
to  the  cognitive  school  of  psychology,  dispute  the 
effect  of  reinforcement.  Robert  Anderson,  theorized 
that  the  assumption  that  KCR  was  s  reinforcer  was  based 
on  an  analogy  to  food  or  water  deprived  animals.  If 
KCR  was  reinforcing,  then  a  delay  in  KCR  should 
interfere  with  learning.  Research  by  the  author  and  by 
other  investigators  showed  delays  improved  retention 
(R.  Anderson,  Kulhavy,  &  Andre,  1972).  Other 
researchers  could  not  confirm  the  reinforcement  aspects 
of  feedback  and  did  find  that  feedback  acted  primarily 
to  correct  errors  (Guthrie,  1971;  Roper,  1977). 

Many  cognitive  psychologists  embrace  the  belief 
that  some  feedback  does  reinforce,  but  informational 
feedback  primarily  serves  to  help  students  locate 
errors  and  provides  information  to  help  learners 
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correct  their  errors  (J.  Carter,  1984;  Shimmel,  1983). 
In  fact  to  some  researchers,  the  number  of  errors 
corrected  is  directly  related  to  the  amount  of 
information  in  the  feedback  (Roper,  1977). 

The  possible  forms  of  informative  feedback  can 
range  from  simple  to  complex.  The  most  basic  provides 
the  learner  only  with  information  about  the  correctness 
of  the  response  and  can  extend  to  the  point  where  the 
amount  of  information  surpasses  the  information 
provided  in  the  original  instruction.  Several 
researchers  have  provided  generally  accepted 
definitions  of  the  levels  of  informational  feedback: 
(a)  Knowledge  of  Response  (KOR),  which  informs  the 
learner  whether  the  selected  response  was  correct;  (b) 
Knowledge  of  Correct  Response  (KCR) ,  which  informs  the 
learner  of  the  correct  response;  and  (c)  Knowledge  of 
Correct  Response  plus  additional  information  which 
details  why  the  correct  answer  is  correct  or  why  the 
incorrect  answer  is  not  correct  (Gilman,  1969a). 

Gilman  (1969a,  1969b)  calls  the  later  type  of  feedback 
Response  Contingent  feedback.  Several  other  variations 
exist:  (a)  KCR  plus  an  explanation  of  why  the  selected 
incorrect  response  was  wrong  or  why  the  correct  answer 
was  correct;  (b)  KOR  plus  information  explaining  why 
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the  incorrect  response  was  wrong  or  why  the  correct 
answer  was  correct;  and  (c)  hints  which  direct  the 
learner  to  the  correct  answer  based  on  stimulating 
recall  of  information  already  provided  in  the  initial 
instructional  sequence.  The  research  literature 
provides  comparisons  of  the  effects  of  varying 
combinations  of  the  different  levels  of  informative 
feedback.  Typically,  enough  variation  exists  in  the 
combinations  of  the  levels  of  feedback,  even  without 
considering  other  variables,  to  preclude  accurate 
comparison  of  results  from  the  study.  In  several 
different  studies  which  compared  an  explanation  with 
KCR  and  KOR,  researchers  found  that  the  explanation  was 
best,  followed  by  KOR  and  KCR  (V.  Cohen,  1985;  Gilman, 
1969a,  1969b;  Kulhavy,  1977;  Roper,  1977). 

Different  levels  of  feedback  seem  to  effect 
learning  differently,  relative  to  the  different  types 
of  learning  outcomes.  Grant,  McAvoy,  and  Keenan  (1982) 
found  that  feedback  explanations  which  focused  on  the 
relevant  critical  attribute  of  a  concept  was  the  most 
effective  feedback  in  concept  learning. 

Other  factors  or  levels  of  feedback  have  been 
examined.  For  example,  hints  have  been  the  focus  of 
instructional  design  and  feedback  research.  Jay 
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(1983),  in  his  recommendations  for  feedback  in 
courseware  design,  proposed  the  use  of  hints  to  sharpen 
a  close  response  or  to  lead  the  learner  to  the  answer 
without  outright  giving  the  answer.  When  discussing 
CAI  instructional  design  considerations.  Smith  and 
Boyce  (1984)  recommend  the  use  of  clues  (hints)  in 
feedback  which  could  be  either  learner  requested  or 
provided  automatically  by  the  software.  Feng  and 
Reigeluth  (1983)  compared  hints,  KCR,  and  KOR.  A  hint 
was  information  designed  to  help  the  learner  select  the 
correct  answer  based  upon  the  instructional  information 
already  presented  in  the  software.  They  found  that 
hints,  then  KCR,  followed  by  KOR,  tended  to  elicit  the 
best  posttest  scores. 

Another  possibility  in  the  practice  and  feedback 
section  of  CAI  which  received  some  attention  was  to 
provide  the  student  a  second  attempt  to  answer  the 
question  after  an  error  (Noonan,  1984;  Smith  &  Boyce, 
1984).  Again,  when  discussing  feedback  in  the 
instructional  design  of  CAI,  Smith  and  Boyce  (1984) 
state  that  one  of  the  values  of  the  CAI  environment  was 
that  learners  could  be  allowed  several  attempts  after 
incorrectly  answering  a  question.  Noonan  (1984) 
investigated  the  impact  of  a  second  attempt  after  two 
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different  types  of  feedback  when  learning  procedural 
objectives  in  a  CAI  program. .  Noonan  found  no 
significant  differences  between  groups.  However,  he 
recognized  that  current  statistical  methods  do  not 
evaluate  the  different  posttest  successes  relative  to 
an  improvement  based  upon  treatment.  Again,  the 
research  is  not  consistent  because  of  the  wide  variety 
of  uncontrolled  and  dissimilar  secondary  variables 
which  do  impact  results.  Many  of  these  research 
studies  conflict  with  on  another,  leaving  the  educator 
to  choose  on  result  or  the  other,  thinking  the  studies 
examined  the  same  questions  under  similar  conditions 
and  received  contradicting  results.  It  appears  more 
work  needs  to  be  done  in  the  research  of  feedback. 

The  diversity  of  the  research  results  on  feedback 
is  potentially  confusing.  Four  factors,  when  combined, 
highlight  the  differences  in  research  results.  The 
function  of  feedback,  the  taxanomic  level  of  the 
learning  objective,  the  focus  on  correctness  and  the 
integration  of  learning  theory  and  cognitive  memory 
theories  all  play  a  part  in  determining  the  most 
appropriate  feedback  to  use.  The  general  recognition 
that  feedback  has  two  functions,  error  correction  and 
confirmation  or  reinforcement,  (Kulhavy,  1977;  Roper, 
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1977)  is  one  factor.  Researchers  often  do  not  clarify 
which  function  they  are  evaluating,  and  they  seldom 
design  materials  and  dependant  measures  which 
differentiate  between  the  two  factors. 

A  second  factor  which  v»ill  help  clarify  the 
results  is  recognition  of  the  fundamental  relationship 
between  the  feedback  and  the  learning  objectives 
(Sales,  1988;  Maher,  1985;  Schimmel ,  1983;  Reiser  &  R. 
Gagne,  1982;  Snyder  &  Land,  1988)  and  the  subseguent 
design  and  report  of  research  project  with  that 
relationship  clearly  identified  and  controlled.  For 
example,  the  cognitive  level  of  the  objective  should 
have  significant  impact  on  feedback.  Schimmel  (1983) 
and  Reiser  and  Gagne  (1982)  found  that  more 
informational  feedback  was  increasingly  helpful  in 
higher  levels  of  cognitive  material.  Therefore,  the 
feedback  used  to  successfully  learn  a  new  concept 
should  be  different  than  the  feedback  use  to  learn  new 
verbal  information. 

The  next  factor  emphasizes  the  need  to  focus  on  the 
correctness  of  the  answer.  The  researchers  in  verbal 
learning  (Kulhavy  et  al.,  1985;  Siegel  &  Misselt,  1984) 
and  those  in  concept  learning  (Shoen,  1972;  Gilman, 
1969b)  identified  informational  feedback  to  be  more 
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beneficial  when  expressed  as  a  condition  of  the 
correctness  of  the  right  answer.  Feedback  which 
concentrates  on  the  student ^s  error  may  have  leave  the 
student  thinking  about  the  error;  this  subsequently 
increases  the  likelihood  that  they  remembered  the 
incorrect  solution.  Going  deeper  into  the  theories, 
Thorndike's  (1932)  Law  of  Exercises  and  John  Anderson's 
(1976,  1983a)  theory  of  reinforcement  and  strengthening 
propositions  could  apply.  The  proposition,  which 
represented  the  incorrect  solution  that  the  learner 
selected  was  activated  and  consequently  strengthened, 
while  the  correct  proposition  was  not  strengthened. 

This  would  also  partially  explain  the  success  of  KCR  in 
verbal  learning,  the  correct  proposition,  restated  in 
the  KCR,  was  strengthened.  The  use  of  a  second  attempt 
also  takes  advantage  of  the  same  strengthening  of  the 
correct  propositional  bonds. 

A  final  factor  integrates  the  ideas  of  Robert  Gagne 
(1965a,  1965b,  1987),  Tennyson  (1980,  1984),  and  J. an 
Anderson  (1974,  1983a,  1984)  and  provide  a  theoretical 
explanation  which  integrates  the  different  results 
regarding  concept  learning.  There  are  differences  in 
how  verbal  information  and  subsequent  concepts  are 
stored  (J.  Anderson,  1974,  1983a,  1984).  The 
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integration  of  their  ideas,  to  some  degree,  is  a 
building  block  approach,  just  as  each  researcher  built 
on  the  developments  of  his  predecessors,  and  just  as 
Robert  Gagne  (1965a)  explains  that  learning  verbal 
information  is  necessary  for  learning  concepts.  Verbal 
learning,  memorization  of  terms  or  words,  for  example, 
is  the  foundation  for  a  concept.  This  is  also 
consistent  with  taxonomies  of  the  cognitive  domain 
being  proprietary  and  hierarchical .  An  abstract 
concept  might  be  learned  through  attribute  listing 
(Tennyson,  1980,  1984;  Tennyson  &  Park,  1980). 

In  egrating  the  ideas  of  Tennyson,  Robert  Gagne,  and 
John  Anderson  might  provide  a  clearer  view  of  abstract 
concept  learning.  The  verbal  information,  stored  as  a 
proposition  which  forms  a  propositional  network  (J. 
Anderson,  1974.  1983a,  1984;  R.  Gagne,  1987).  Those 
propositional  networks  are  arranged  to  build  a 
production  (J.  Anderson,  1974,  1983a,  1984)  which  is 
the  attribute  list  described  by  Tennyson  (1980,  1984) 
necessary  for  concept  acquisition,  especially  for 
abstract  concepts.  Informational  feedback  provides 
information  to  correct  the  production.  Informational 
feedback  can  add,  remove,  or  rearrange  the  order  of  the 
propositions  in  the  production,  especially  while  still 
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located  in  STM.  While  KCR  is  enough  information  to 
correct  the  error  in  a  proposition,  or  the  verbal 
information,  it  may  not  be  enough  to  correct  the  order 
of  the  propositions  in  the  production  which  in  turn 
describes  a  concept.  This  would  explain  the  reported 
success  of  KCR  in  concept  learning.  When  the  error  is 
in  the  proposition,  the  verbal  information  level,  and 
not  the  production,  KCR  is  enough  feedback. 
Informational  feedback  goes  beyond  KCR  because  it 
provides  the  corrective  information  necessary  to 
correct  the  production,  and  ordered  propositional 
network.  This  seems  most  applicable  to  abstract 
concepts,  but  the  integration  of  theories  also  explains 
how  concrete  concepts  may  be  created  in  human  memory. 

In  addition,  the  descriptions  of  concrete  concepts 
versus  abstract  concepts  could  be  reflected  in  John 
Anderson's  alternate  method  for  storage  in  memory, 
images.  Consequently,  a  reason  for  the  success  of 
prototyping  theory,  is  that  a  verbal  picture  or  an 
actual  picture,  the  prototype  is  used  to  develop  a 
concrete  concept  in  memory. 
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Method 


Subjects 

The  subjects  were  173  freshman  college  students  at 
the  United  States  Air  Force  Academy.  Entry 
requirements  place  these  students  in  the  top  quarter  of 
the  class  of  college  freshman  nationally. 

Materials 

The  computer  program  used  to  present  the  lesson  was 
a  researcher-designed  program  based  on  a  linear  format. 
The  program  was  compiled  in  Turbo-Pascal  for  a  80286 
microprocessor  based  computer  system  with  51 2K  RAM,  EGA 
screen,  and  two  360K  floppy  disk  drives  or  one  hard 
disk  with  720K  of  memory  free.  No  other  materials  were 
required. 

Three  fundamental  concerns  prompted  the  selection 
of  the  subject.  A  major  factor  in  the  study  was  the 
need  to  teach  a  new  subject  to  the  learner,  but  one 
that  they  were  motivated  to  study.  Secondly,  the 
learning  objectives  were  from  the  concept  level  of 
Robert  Gagne's  (1974)  taxonomy.  It  was  also  preferable 
that  prerequisite  fundamental  knowledge  or 
discriminations  necessary  to  learning  the  concept  were 
covered  in  the  CAI  lesson  so  errors  in  prerequisite 
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knowledge  did  not  impact  on  concept  attainment. 

Finally,  a  large  number  of  learners  was  needed,  as  the 
collectable  data  was  based  on  feedback  provided  on 
error.  Unless  the  material  was  unreasonably  difficult 
and  didn't  reflect  the  level  of  materials  typically 
expected  for  the  academic  group.  Without  a  large 
group,  there  would  not  have  been  many  errors,  hence 
there  would  have  been  little  data  to  analyze. 
Navigation,  a  required  course  for  all  students,  as  a 
good  candidate.  Few  high-school  students  study 
navigation 

A  linear  program  format  does  not  take  advantage  of 
the  computer's  capability  to  branch  within  a  program 
and  to  repeat  a  block  of  instruction  based  upon  learner 
request.  However,  computer  based  software  was 
necessary  to  evaluate  the  media  and  it  provided  easy 
control  for  time-on-task,  access  to  knowledge  level 
information,  number  of  attempts,  and  several  other 
factors  which  might  confound  results. 

Procedures 

The  experimental  procedure  followed  a  pretest, 
intervention,  and  posttest  sequence.  The  pretest  and 
delayed  posttest  were  conducted  as  a  computer  base 
exercise,  identical  in  format  to  the  practice  questions 


299 


and  the  immediate  posttest.  The  intervention 
containing  the  treatment  and  the  immediate  posttest  was 
conducted  as  a  CAI  lesson  in  a  computer  laboratory. 

The  participants  were  administered  a  computer  based 
pretest.  The  students  met  in  their  regularly  scheduled 
classroom  and  then  were  directed  to  the  lab,  where  the 
pretest  was  conducted  at  the  beginning  of  the  regularly 
scheduled  class.  The  participants  were  randomly 
assigned  to  one  of  six  treatment  groups,  stratified  on 
pretest  score.  While  all  students  completed  the 
intervention,  however,  only  subjects  with  pretest 
scores  less  than  eighty  percent  were  considered  in  the 
analysis. 

When  subjects  arrived  at  the  computer  lab  for 
instruction  a  system  was  available  for  every  subject. 
Each  system  contained  an  identical  copy  of  the 
instructional  software  and  the  pretest  score-treatment 
group  data  base  on  the  hard  disk.  The  system  accessed 
the  data  base  and  presented  the  instruction  with  the 
appropriate  randomly  assigned  treatment.  The 
instructional  software  recorded  the  start  time  from  the 
computer.  Once  the  student  has  completed  the 
instruction  the  computer  recorded  the  subject's 
completion  time  as  well  as  the  stop  and  start  time  for 
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each  objective.  The  student  was  presented  with  the 
posttest.  Once  complete,  the  software  saved  the  SSAN, 
pretest  score,  treatment  group,  total  time,  time  by 
objective,  responses  with  an  identifier  for  a  correct 
or  incorrect  response  for  both  first  and  second  attempt 
if  applicable,  and  posttest  score  in  the  data  base  on 
the  hard  disk. 

The  students  attended  their  next  regularly 
scheduled  class.  At  the  beginning  of  the  class,  the 
students  were  again  moved  to  the  lab  and  received  an 
unannounced  posttest  on  the  computer.  It  followed  the 
same  format  as  the  pretest,  the  instruction,  and  the 
immediate  posttest.  Their  delayed  posttest  scores  were 
added  into  the  data  base. 

Analysis 

Results 

For  statistical  analysis,  ANCOVA  was  used  with 
attempts  and  level  of  feedback,  the  two  independent 
variables,  as  categories.  The  percentage  of  first 
attempt  errors  on  the  practice  questions  was  the 
covariate.  For  level  of  feedback  on  the  immediate 
posttest,  the  F-ratio  was  not  significant  at  p  <  0.05. 
In  contrast,  level  of  feedback  on  the  delayed  posttest 
was  significant  at  p  <  0.01.  For  one  versus  two 
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attempts  on  the  immediate  posttest  and  on  the  delayed 
posttest  the  results  were  significant  at  p  <  0.006  and 
p  <  0.046  respectively. 

Discussion 

Significant  results  indicate  that  a  second  attempt 
to  answer  practice  questions  which  were  missed  on  first 
attempt  has  an  impact  on  acquisition  and  retention  of  a 
new  concept.  This  confirms  the  intuition  of  many 
classroom  teachers  and  instructional  designers,  as  well 
as  the  theoretical  foundation  established  by  Thorndike 
(1932)  in  his  Law  of  Exercises  and  John  Anderson's 
(1976,  1983c)  theory  of  reinforcement  and  strengthening 
propositions.  These  findings  partially  support  the 
design  recommendations  of  Smith  and  Boyce  (1984)  who 
stated  that  one  of  the  values  of  the  CAI  environment 
was  that  learners  could  be  allowed  several  attempts 
after  incorrectly  answering  a  question.  It  would  be 
inappropriate  to  extend  the  findings  of  this  study  past 
a  second  attempt,  but  it  showed  significant  results  for 
one  additional  attempt  to  answer  a  practice  question  in 
concept  learning. 

Like  second  attempt,  the  increase  in  the  level  of 
informational  feedback  showed  significant  effects  on 
retention.  This  again  supports  the  theoretical 
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framework  of  this  study  and  complements  the  research  on 
feedback  by  Gilman  (1969a)  in  which  increased 
information  was  identified  as  beneficial  to  retention 
and  by  Feng  and  Reigeluth  (1983)  who  focused  on  hints. 

Unlike  the  effects  found  on  retention,  as  measured 
by  the  delayed  posttest  described  above,  the  effect  of 
levels  of  feedback  on  concept  acquisition,  as  measured 
by  an  immediate  posttest,  were  not  significant. 

Further  examination  of  the  data  showed  that  the 
posttest  mean  of  KORl  was  86.25;  for  KORIl  it  was 
86.714;  and  for  KORIHl  it  was  83.037.  The  results  were 
almost  directly  contrary  to  prediction,  however  the 
results  were  similar  to  those  of  Bodes  (1985)  and 
Kulhavy,  et  al.  (1985). 

Interviews  with  the  students  in  each  treatment 
group,  may  have  identified  the  reason  for  these 
seemingly  anomalous  results.  Students  in  the  treatment 
group  which  received  only  Knowledge  of  Results  (KOR) 
feedback  were  very  unhappy  that  they  did  not  get  any 
extensive  feedback.  When  asked  why  they  were  upset, 
they  stated  that  they  had  to  work  hard  in  the 
beginning,  when  the  initial  instruction  was  being 
presented,  so  they  could  do  well  on  the  exams.  The 
students  in  the  Knowledge  of  Results  plus  Information 
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(KORI)  and  Knowledge  of  Results  plus  Information  plus 
Hint  (KORIH)  groups  said  they  liked  the  feedback 
because  they  could  breeze  through  the  initial 
instruction,  thus  saving  time.  If  they  made  an  error, 
the  feedback  would  save  them  because  it  focused  on  the 
attributes  of  the  concept  they  did  initially  acquire. 
The  ability  to  save  time  was  emphasized  by  the  students 
who  received  KORIH.  The  hint  helped  them  focus  on  the 
part  of  the  correct  answer  they  had  missed. 

R.C.  Anderson,  Kulhavy,  and  Andre  (1971) 
identified  this  as  "short-circuiting”  instruction  with 
feedback.  In  their  study  which  used  KCR  after  teaching 
a  medical  concept,  the  researchers  emphasized  two 
possible  explanations  for  the  potential  failure  of  KCR. 
The  first  was  short-circuiting  the  instruction.  The 
students  were  able  to  find  the  correct  response  to  the 
questions  in  the  feedback,  so  they  chose  any  answer, 
read  the  feedback,  and  selected  the  correct  response. 
Their  second  idea  highlighted  a  feature  of  human  nature 
which  may  have  motivated  the  short-circuiting,  that  is, 
the  law  of  least  effort,  which  functions  when  students 
are  tired  or  under  pressure.  When  students  knew  KCR  was 
available,  they  studied  less  carefully  or  spent  less 
time  in  the  primary  instruction. 
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In  the  case  of  this  study,  the  short-circuiting  of 
the  instruction  with  the  feedback  may  have  been 
counter-productive  to  accurate  results.  If  the 
students'  objective  was  to  answer  the  test  properly  but 
as  easily  as  possible,  the  feedback  was  ideal  for 
short-circuiting  the  instruction.  The  learner  could 
move  quickly  through  the  primary  instruction,  selected 
their  best  choice,  and  the  feedback  would  identify  and 
correct  the  misunderstanding  if  an  error  was  made.  The 
feedback  in  the  ungraded  practice  questions  focused  on 
the  correct  attributes  of  the  correct  answer  to  the 
practice  questions .  The  hint  was  even  more  focused  on 
the  correct  answer.  The  practice  questions  were 
similar  to  the  posttest,  so  the  students  could  expect 
do  well  on  the  posttest.  Second  attempt  students  were 
then  able  to  confirm  their  selection.  The  short- 
circuiting  may  have  been  a  learning  strategy  they 
developed  in  the  past  to  compensate  for  poor 
instruction  or  to  save  time.  This  student  sub¬ 
population  was  selected  to  attend  the  Academy  because 
of  its  record  of  academic  capability  and  its  success. 
The  possibility  exists  that  these  subjects  short- 
circuited  the  instruction,  hence  the  results,  in  their 
attempt  to  ensure  success  with  relative  ease. 
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Instructional  Design 

The  results  of  this  study  could  be  interpreted  to 
mean,  that  if  one  is  a  good  designer,  and  one  wants  his 
or  her  students  to  do  the  best  possible  job  of 
acquiring  and  retaining  a  new  concept,  one  should  use 
the  highest  level  of  feedback,  knowledge  of  results 
plus  information  plus  a  hint,  with  a  second  attempt. 

The  use  of  second  attempt  is  a  simple  and  effective 
tool  to  improve  acquisition  and  retention  and  should  be 
included.  However,  the  more  sophisticated  the  level  of 
informational  feedback,  the  more  work,  hence  higher 
development  costs,  are  involved.  Adding  higher  levels 
of  instructional  feedback  involves  significantly  more 
work  than  including  a  second  attempt. 

If  the  analysis  of  the  learner  in  the  design 
process  shows  high  end  academic  achievement,  it  may  be 
appropriate  to  let  the  learner  choose  the  level  of 
feedback,  particularly  if  retention  is  not  a  concern. 

In  fact,  since  the  student  may  short-circuit  the 
instruction  with  the  feedback,  and  the  learner  is  going 
to  spend  approximately  the  same  time  with  either 
choice,  it  may  be  appropriate  based  on  other  external 
objectives  to  include  only  second  attempt  in  the 
practice  question  feedback. 
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Summary 


This  study  examined  the  effects  of  two  variables, 
level  of  feedback  and  number  of  attempts  on  the 
practice  questions  in  a  CAI  lesson.  The  levels  of 
feedback  ranged  from  KOR,  through  KOR  plus  additional 
information,  to  the  inclusion  of  a  hint.  Each  level 
increased  the  level  of  informational  feedback,  and  the 
hint  also  provided  an  alternate  cue  for  the  recall  of 
the  propositional  networks.  In  addition,  the 
information  in  the  feedback  focused  on  the  correct 
attributes  of  the  subject,  a  concept.  The  feedback  did 
not  focus  on  why  the  incorrect  selection  was  wrong. 

Half  of  the  students  in  each  group  also  received  a 
second  attempt  to  answer  the  practice  question. 

Student  acquisition  and  retention  was  measured  on  an 
immediate  posttest  and  a  delayed  posttest.  The  fact 
that  feedback  was  initiated  on  student  error 
necessitated  the  inclusion  of  a  covariate,  the 
percentage  of  errors  on  first  attempt  practice 
questions,  in  the  computation  of  analysis  of  variance. 
The  results  supported  the  hypothesis  that  a  second 
attempt  to  answer  the  practice  question  increased  both 
acquisition  and  retention.  The  increased  levels  of 
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feedback,  including  hint,  were  supported  as  an  aid  for 
retention,  but  did  not  aid  acquisition. 

Instructional  designers  are  able  to  easily  add  a 
second  attempt  after  incorrectly  answered  practice 
questions  to  benefit  acquisition  and  retention.  While 
previously  applied  by  instructional  designers  due  to 
intuitive  understanding  of  successful  designs,  the 
significant  results  of  this  study  provide  support  for 
regularly  incorporating  second  attempt  into  lessons. 

The  addition  of  increased  levels  of  informational 
feedback  should  be  examined  on  a  case-by-case  basis,  as 
adding  complex  feedback  may  significantly  increase 
development  costs.  If  retention  is  a  consideration, 
extended  informational  feedback  should  be  included 
based  on  the  significant  results  of  this  study.  The 
examination  of  the  effect  of  the  levels  of 
informational  feedback  on  concept  acquisition  should  be 
confirmed  with  another  research  study,  followed  by 
examination  of  both  independent  variables  from  this 
study  in  rule  learning  and  problem  solving  lessons. 
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QUESTIONS  AND  FEEDBACK  IN  COMPUTER-ASSISTED  INSTRUCTION 


Major  Woodrow  T.  Hawley, PhD. 
Chief,  Military  Training  Division 
Commandant  of  Cadets, USAF  Academy,  CO 


This  study  investigated  the  effects  that  matching  level  of 
feedback  with  level  of  question  has  on  academic  achievement  as 
measured  by  an  immediate  and  delayed  posttest.  The  level  of 
feedback  and  level  of  questions  were  combined  to  form  four 
treatment  groups:  low-level  questions  with  KCR  feedback,  low- 
level  questions  with  KCR+E  feedback,  high-level  questions  with 
KCR  feedback,  and  high-level  questions  with  KCR+E  feedback. 

Subjects  were  undergraduate  students  enrolled  in  a 
navigation  course.  There  were  no  significant  differences  between 
groups  on  the  immediate  and  delayed  posttest  scores  when  the 
level  of  feedback  matched  the  level  of  question.  However, 
subjects  who  received  KCR  feedback  after  low-level  questions 
scored  significantly  higher  on  the  immediate  posttest  than 
subjects  who  received  KCR+E  feedback  after  high-level  questions. 
Additionally,  subjects  who  received  low-level  questions  scored 
significantly  higher  on  the  immediate  and  delayed  posttests  than 
subjects  who  received  high-level  questions. 


Major  Woodrow  T.  Hawley  received  a  degree  of  Bachelor  of  Arts 
from  South  Dakota  State  University  and  was  commissioned  a  Second 
Lieutenant,  United  States  Air  Force,  in  December  1975.  He 
entered  the  United  States  Air  Force  in  October  1976.  In  the 
following  years  he  served  as  a  Missile  Launch  Officer  and  then 
Instructor  at  the  United  States  Air  Force  Academy,  where  he  is 
currently  an  Assistant  Professor  and  Chief  of  the  Military 
Training  Division.  He  received  a  degree  of  Master  of  Arts  in 
June  1981  from  the  University  of  Northern  Colorado.  In  June  1990 
he  received  a  PhD  in  Education  from  the  University  of  Texas  at 
Austin . 
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CCK)PERATIVE  LEARNING  IN  COMPUTER  ASSISTED  INSTRUCTION 


Major  Michael  M.  Whyte,  PhD. 

Chief,  Military  Studies  Division 
Commandant  of  Cadets,  USAF  Academy,  CO 

A  significant  portion  of  instructional  research  has 
focused  on  individual  instruction  and  success.  However,  the 
group  approach  may  provide  solutions  to  educational  problems 
of  efficiency,  cost,  limited  resources  and  instructors. 

This  study  was  to  determine  whether  paired/cooperative 
computer  assisted  instruction  (CAI)is  as  effective  as  an 
individualistic  approach.  This  study  also  examined  the 
interactive  effects  of  individual  cognitive  style  on 
paired/cooperative  CAI .  Three  different  pairings  were 
analyzed  according  to  affective  and  achievement  measures. 

Of  86  students,  57  were  randomly  assigned  to  the 
paired/cooperative  treatment  and  29  were  assigned  to  the 
individual  instruction  treatment. 

No  significant  difference  existed  between  the  mean 
posttest  scores  of  participants  who  worked  individually  and 
those  who  worked  in  pairs.  The  manner  in  which  individuals 
are  paired  by  cognitive  style  not  only  made  a  significant 
difference  in  achievement  test  scores  but  also  in  affective 
interaction.  Groups  containing  at  least  one  field 
independent  student  significantly  outperformed  groups 
containing  two  field  dependent  learners.  Field  dependent 
students,  therefore,  benefited  significantly  when  paired 
with  a  field  independent  student. 


Major  Whyte  graduated  from  the  USAF  Academy  with  a  degree  in 
International  Affairs  and  American  Politics.  He  is  also  a 
graduate  of  the  University  of  Southern  California,  with  a 
Master  of  Science  degree  in  Education  and  a  Doctor  of 
Philosophy  in  Instructional  Technology.  After  initial 
aircrew  assignments  as  a  Special  Air  Mission  Instructor 
Pilot  and  as  a  Chief  of  International  Programs,  Pentagon,  he 
was  assigned  to  USAFA.  In  addition  to  his  current 
responsibilities,  he  has  taught  five  different  university 
courses  and  served  as  a  Squadron  Training  Officer. 


COOPERATIVE  LEARNING  IN  COMPUTER  ASSISTED  INSTRUCTION 


INTRODUCTION 


Many  look  to  emerging  technologies,  such  as  computers, 
as  our  "new  hope"  for  improvement  in  the  quality  of 
education.  The  amount  of  substantive  research,  however, 
involving  how  individuals  or  pairs  learn  on  computers  is 
relatively  small.  We  know  little  about  how  learners  with 
individual  cognitive  differences  react  to  various  teaching 
methods  such  as  paired  versus  individual  computer-assisted 
instruction  (CAI).  Educators  need  to  learn  more  about  the 
interaction  of  students  and  delivery  systems  in  order  to 
construct  more  effective  models  and  tools.  Researchers  at 
the  University  of  Southern  California  found  that  individuals 
with  certain  cognitive  styles  often  react  differently  to 
paired  learning  situations.  They  also  found  that  the 
pairing  of  certain  individuals  can  have  significant  effects 
on  both  student  attitude  and  achievement  (Whyte,  Knirk, 

Casey  &  Willard,  1990-91).  This  paper  contains  substantive 
information  from  not  only  the  previously  cited  article  but 
also  from  the  author's  dissertation  research.  Much  more 
research  is  needed  in  the  area  of  pairing  students  by 
individual  differences. 


CCX3NITIVE  STYLE 


In  his  article,  Messick  (1984)  defines  cognitive  style 
as  "characteristic  self-consistencies  in  information 
processing  that  develop  in  congenial  ways  around  underlying 
personality  trends."  He  also  states  that  cognitive  styles 
are  "intimately  inter-woven"  with  motivational,  affective 
and  temperamental  structures  to  produce  total  personality. 
Perhaps  the  most  prolific  writer  in  the  area  of  cognitive 
styles  was  Herman  A.  Witkin.  As  part  of  his  extensive 
research,  Witkin,  Moore,  Goodenough  and  Cox  (1977) 
determined  that  cognitive  styles:  1)  deal  with  the  "form"  of 
cognitive  activity,  not  its  content  (e.g.,  thinking, 
perceiving,  problem  solving,  etc.);  2)  are  "persuasive 
dimensions"  in  that  they  are  a  feature  of  not  only 
personality  but  also  cognition;  3)  are  stable  over  time;  and 
4)  are  also  bipolar.  In  other  words,  being  on  one  end~of  a 
cognitive  style  dimension  may  be  useful  in  some 
circumstances  while  not  in  others.  This  aspect  is  in 
contrast  to  intelligence,  for  example,  where  "more"  is 
always  "better."  Cognitive  style  is,  therefore,  concerned 
with  how  an  individual  processes  information.  This  includes 
any  process  which  acquires  knowledge  (e.g.,  memory, 
perception,  thought,  and/or  problem  solving).  Cognitive 
styles  are  also  not  likely  to  change  with  training  (Ausburn 
&  Ausburn,  1978).  Training,  therefore,  should  be  adapted  to 


the  learner  rather  than  the  learner  to  training.  Cognitive 
style  is  not  a  single  entity.  Most  educational 
psychologists  recognize  nine  to  eleven  major  dimensions  of 
cognitive  style  including:  ref lectivity/impulsivity ; 
scanning;  categorizing;  and  field  independence/f ield 
dependence  (Carrier  &  Jonassen,  1988). 


OVERVIEW  OF  FIELD  INDEPENDENCE/FIELD  DEPENDENCE 

According  to  Rasinski  (1983)/  field  independence/f ield 
dependence  is  "by  far"  the  most  researched  of  all  cognitive 
styles.  It  also  seems  to  have  the  greatest  potential  for 
application  to  educational  problems.  Most  research  in  this 
area  began  in  the  early  1950 's  and  1960 's.  Herman  A. 

Witkin,  Donald  R.  Goodenough  and  Philip  K.  Oltman  (1986) 
have  produced  most  of  the  substantive  research  in  this  area 
in  the  last  30  years.  According  to  Willard  (1985),  this 
dimension  is  concerned  with  a  learner's  ability  to  "perceive 
a  part  of  a  stimulus  as  discrete  from  its  surroundings 
through  active  and  analytic  as  opposed  to  passive  and  global 
processes".  It  is  also  significant  to  note  that  the  FI/FD 
dimension  is  "bipolar".  According  to  Witkin  and  Goodenough 
(1977),  "each  of  the  contrasting  cognitive  styles  has 
components  that  are  adaptive  to  particular  situations, 
making  the  dimension  value  neutral". 


Specific  Characteristics  of  Field  Independence  (FI) 


Canino  and  Cicchelli  (1988)  define  FI  individuals  as 
those  who  are  capable  of  perceiving  items  as  discrete  from 
background  or  field.  They  also  learn  better  when  they  are 
allowed  to  develop  their  own  strategies  in  problem-solving 
nonsocial  domains.  Rosenberg,  Mintz  &  Clark  (1977)  compiled 
research  findings  on  this  particular  dimension  which  dealt 
specifically  with  learning  preferences  and  abilities.  They 
found  that  FIs:  1)  play  a  much  more  participant  or  active 
learning  role;  2)  learn  more  "efficiently"  in  conditions  of 
self  motivation;  3)  learn  more  effectively  without 
performance  feedback;  4)  do  not  seem  to  need  an  externally 
provided  structure;  5)  are  more  attentive  to  "nonsalient 
attributes  in  concept  learning  tasks";  and  6)  t<*nd  to  favor 
lectures  (i.e.,  expository  methods)  (Rosenberg,  Mintz  & 
Clark,  1977). 

Specific  Characteristics  of  Field  Dependence  (FD) 

These  learners  are  defined  as  those  who  are  not  as  able 
to  separate  "elements  from  their  surroundings".  They 
experience  their  environment  more  globally  and  usually 
accept  the  organization  provided  by  the  "perceptual  field". 
FDs  also  prefer  to  interact  with  a  teacher  and  tend  to  learn 
better  v;ith  structure  (Canino  and  Cicchelli,  1988). 


Rosenberg,  Mintz  &  Clark  (1977)  also  compiled  research 
findings  on  the  FD  dimension  which  dealt  specifically  with 
learning.  They  found  that  FDs:  1)  tend  to  learn  socially 
relevant  material  better  than  FIs;  2)  usually  perform  a 
spectator  or  passive  learning  role;  3)  are  more  affected 
than  FIs  by  negative  reinforcement;  4)  tend  to  be  more 
influenced  by  the  opinions  of  others  or  authority  figures; 

5)  possess  lower  "performance  expectations";  6)  perform 
stereotyped  roles;  and  7)  prefer  discussions  (i.e., 
interactive  met’'Gds). 

In  ot’.f  words,  FIs  and  FDs  are  almost  complete 
opposit-js  in  learning  preferences  and  abilities.  FI  students 
are  analytical  and  FD  students  are  global. 

PAIRED/C(X)PERATIVE  VERSOS  INDIVIDUAL  INSTRUCTION 

Numerous  studies  conducted  since  the  mid-70's  have  also 

shown  that  cooperative  teaching  methods  can  be  not  only  more 

effective  but  also  more  efficient  than  traditional  metl^ods 

(Johnson,  Johnson  &  Stanne,  1986).  Valient,  Glachan  and 

Emler  (1982)  cite  Piaget's  writings  which  place  a 

significant  importance  on  peer  interaction.  Piaget  believed 

that  social  interaction  not  only  provides  learners  with 

contrasting  viewpoints  and  judgements,  but  also  encourages 

them  to  support  their  own.  Valiant’s  own  research  seems  to 

support  this  as  she  found  that  cooperative  learning  was  not 

only  more  effective  but  also  more  efficient  in  teaching 
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children  classification  skills  (Valient,  Glachan  and  Emler, 
1982).  Several  other  studies  have  also  provided  both 
interesting  and  significant  conclusions  in  the  area  of 
paired/cooperative  instruction.  Noble  (1967)  conducted  one 
of  the  first  experiments  which  studied  the  attitudes  of 
students  who  worked  in  pairs  versus  those  who  worked 
individually.  He  found  no  significant  difference  in 
attitudes  between  the  two  groups.  Incidentally,  he  also 
found  that  a  paired  approach  was  a  cost  effective 
alternative  for  programmed  instruction.  In  1978  Cloutier 
and  Goldschmid  (1978)  found  a  significant  educational 
advantage  of  having  peers  interact  during  training.  This 
seemed  to  be  true  even  in  the  absence  of  an  instructor. 
Merely  pairing  students,  however,  may  not  be  the  answer. 

We,  as  teachers,  do  not,  as  yet,  know  what  makes-up  an 
effective  group. 

Johnson,  Maruyama,  Johnson,  and  Nelson  (1981)  also 
found  that  paired/cooperative  learning  promotes  higher 
achievement  than  either  competivive  or  individualistic 
instruction.  Much  more  research  is  needed  in  the  area  of 
pairing  students  by  individual  differences.  The  next  part 
of  this  section  will  deal  with  the  pairing  of  students  by 
cognitive  style. 
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Paired  Learning  and  Cognitive  Style 


An  example  of  a  study  which  paired  students  according 

to  ability  or  cognitive  style  was  conducted  by  Amaria,  Biran 

and  Leith  (1969).  Results  indicated  that  paired  children 

significantly  outperformed  individuals.  In  this  case, 

students  were  paired  according  to  IQ.  Homogeneous  groups 

(i.e.,  those  with  similar  IQs)  outperformed  heterogeneous 

groups  (i.e., those  with  a  mix  of  above  and  below  average 

intellectual  abilities).  Further  findings  also  supported 

the  fact  that  lower  ability  students  were  academically 

stimulated  by  bright  students.  These  bright  students  were 

also  kept  from  "jumping  ahead."  In  another  related  study, 

Fitzgibbons,  Goldberger  and  Eagle  (1965)  supported  the 

notion  that  individuals  with  certain  cognitive  abilities  can 

react  differently  to  paired  learning  situations.  They  found 

that  individuals  designated  as  field  dependent,  for  example, 

are  more  sensitive  to  "people  centered"  environments.  As  a 

result,  they  rely  more  on  help  provided  by  others. 

Goodenough  also  found  that  individuals  with  differing 

cognitive  styles  assume  different  roles  when  participating 

in  paired  instruction.  He  concluded  that  FDs  are  more 

socially  oriented  and  perform  a  more  spectator  role  during 

paired/cooperative  training.  FIs,  on  the  other  hand, 

perform  a  much  more  participatory  role  (Goodenough,  1976). 

In  1986  Bertini,  Pizzamiglio,  and  Wapner  supported  this 

notion  when  they  determined  that  FDs  rely  more  on 
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interpersonal  relationships  and  are  more  "socially 
oriented".  They  pay  much  closer  attention  to  interpersonal 
cues,  show  more  emotional  openness  in  conversations  and 
prefer  to  work  physically  closer  to  others.  FDs  also  tend 
to  particularly  need  other  people's  input  when  faced  with 
ambiguous  situations  (e.g.,  novices  in  an  introductory 
computer  course).  FIs,  on  the  other  hand,  are  not  only 
vastly  more  impersonal,  but  also  are  disinterested  in 
working  with  others  either  emotionally  or  physically.  The 
way  individuals  are  paired  also  seems  to  have  an  impact  on 
the  effectiveness  of  instruction.  Oltman,  Goodenough, 
Witkin,  Freedman,  and  Friedman  (1975)  conducted  one  of  the 
first  studies  which  dealt  with  the  pairing  of  individuals 
according  to  cognitive  style.  Results  showed  that  dyads  of 
"low-differentiation"  (e.g.,  PI  and  FI  or  FD  and  FD) 
reconciled  disagreements  more  often  and  displayed  greater 
■'interpersonal  attraction"  than  dyads  of  "high- 
differentiation  (e.g.,  FI  and  FD)". 

COMPUTER  ASSISTED  INSTRUCTION  (CAI) 

CAI  and  Cognitive  Style 

Recent  educational  research  has  indicated  that 
instruction  can  be  made  more  effective  if  CAI  is  adapted  to 
individual  cognitive  style  (Cheney,  1980).  Post  (1987)  also 
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found  that  FIs  significantly  outperform  FDs  in  courses 
utilizing  computer  assisted  instruction. 

CAI  and  Paired/Cooperative  Instruction 

Studies  show  that  the  use  of  paired/cooperative 

teaching  methods  result  in  both  more  effective  and  more 

efficient  computer  assisted  instruction  (Hooper  &  Hannafin, 

1988).  Ronald  S.  Lemos  (1979)  has  conducted  an  extensive 

amount  of  research  in  the  area  of  cooperative  computing. 

His  research  concluded  that  a  team  learning  approach  was 

significantly  more  cost  effective  in  teaching  COBAL 

programming.  Developing  cost  effective  alternatives  for 

computer  training  is  especially  important  since  70%  of  the 

U.S.  labor  force  depends  on  data  processing  services  (Lemos, 

1978).  There  are  also  several  purely  pragmatic  reasons  for 

conducting  a  team  approach  in  computer  training.  According 

to  Khailany  and  Saxon  (1978),  individual  computer  processing 

is  virtually  non-existant  in  the  commercial  environment. 

Auld,  Lang,  and  Lang  (1981)  warn  against  the  "social 

isolation"  of  individual  computer  users.  According  to  their 

research,  these  "isolates"  have  fewer  opportunities  to 

discuss  their  work  and,  therefore,  may  have  unnecessary 

difficulty  performing  computing  tasks.  Her/his  motivation 

and  attitude  toward  computers  is,  as  a  result,  negatively 

affected.  The  promotion  of  cooperative  groups  permits 

"egoless  programming"  as  programs  developed  are  the  property 
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of  the  group  rather  than  specific  individuals  (Lemos  1979 
and  Shneiderman  1980).  These  group  processes,  according  to 
Shneiderman  (1980),  also  serve  as  a  type  of  group  therapy  as 
they  encourage  cooperation,  endeavor  to  overcome  anxiety  and 
appear  to  build  interdependence.  Small  groups  encourage 
individual  learners  to  "perform  at  higher  levels"  due  to  the 
fact  that  good  work  will  be  recognized. 


PORPOSE  OP  THIS  STUDY 


The  primary  purpose  of  this  study  was  to  determine 
whether  paired/cooperative  computer  assisted  instruction 
(CAI)  is  as  effective  as  an  individualistic  approach.  This 
study  also  examined  the  interactive  effects  of  individual 
cognitive  style  on  paired/cooperati/e  CAI.  Three  different 
pairings  were  analyzed  according  to  affective  and 
achievement  measures  (i.e.,  PI  with  FI,  FD  with  FI,  and  FD 
with  FD) . 


RESEARCH  QUESTIONS 

This  investigation  sought  to  answer  the  following 
questions : 

1.  What  is  the  relationship  between  mode  of 

instruction  (i.e.,  paired/cooperative  and 

individualistic)  and  achievement  after 

computer  assisted  instruction? 
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2.  What  is  the  relationship  between  mocie  of 
instruction  (i.e.,  paired/cooperative  and 
individualistic)  and  attitude  after 
computer  assisted  instruction? 

3.  How  do  various  pairings  of  students, 
according  to  individual  cognitive  style, 

(i.e.,  FI  and  FD)  effect  student 
achievement? 

4.  How  do  various  pairings  of  students, 
according  to  individual  cognitive  style, 

(i.e.,  FI  and  FD)  effect  student  attitude? 

METHODOLOGY 

The  research  design  selected  for  this  study  was  a 

quasi-experimental,  three-group,  posttest-only  design  Each 

of  the  86  participants  of  the  training  portion  of  this 

investigation  was  first  administered  the  Group  Embedded 

Figures  Test  (GEFT) .  The  GEFT  measures  the  cognitive  style 

dimension  of  field  independence  and  field  dependence.  The 

reliability  of  this  test  is  .82  for  both  females  and  males 

[10].  After  this  preliminary  test,  students  were  randomly 

assigned  to  one  of  four  groups.  The  groups  consisted  of  the 

following;  1)  individuals;  2)  a  FI  paired  with  a  PI; 

3)  a  FD  paired  with  a  FI;  and  4)  a  FD  paired  with  a  FD.  The 

independent  variable  used  in  this  experiment  was  the 

particular  mode  of  instruction  given  to  students.  Out  of  86 
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total  students,  57  were  randomly  assigned  to  the 
paired/cooperative  treatment  and  29  students  were  assigned 
to  the  individualistic  treatment.  This  method  of  group 
assignment  is  consistent  with  other  studies  of  this  type 
[8].  Students  were  then  randomly  assigned  to  computers. 

All  students,  regardless  of  instructional  mode,  however, 
received  exactly  the  same  instructional  content.  During  the 
workshop  students  were  taught  a  series  of  DOS  commands  by 
means  of  lecture  and  hands-on  experience.  Individual 
students  were  told  to  work  on  their  own,  while  paired 
students  were  briefed  to  equally  share  "keyboard  time." 

Instruction  included  a  self-paced  computer  tutorial 
which  reviewed  all  of  the  basic  DOS  commands  taught  during 
the  lecture  portion.  The  tutorial,  therefore,  served  not 
only  as  a  review  of  the  material,  but  also  was  an  excellent 
control  for  our  training.  An  additional  twelve  subjects 
made  up  the  Control  Group.  The  primary  dependent  variable 
utilized  in  this  study  was  a  participant's  score  on  a  20 
question  test  on  various  disc  operating  system  commands. 

DOS  commands  included  formatting,  "booting",  renaming  files, 
and  making  and  changing  directories.  Finally,  students  were 
asked  to  complete  a  questionnaire  which  was  tailored  to 
his/her  randomly  assigned  mode  of  instruction. 

A  T-Test  was  used  as  a  comparison  of  pairs'  and 

singles'  achievement  tests.  Analysis  of  the  results  of  the 

three  pairs  of  individual  cognitive  styles  (i.e.,  FI  with 

FI,  FD  with  FI,  and  FD  with  FD)  was  accomplished  by  using  a 
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one-way  analysis  of  variance  (ANOVA) .  Finally,  the  Scheffe 
Procedure  was  used  to  determine  where  the  three  pairs 
significantly  differed  at  the  0.050  level. 


DATA  ANALYSIS 


Table  1  shows  that  no  significant  difference  existed 
between  the  mean  posttest  scores  of  participants  who  worked 
individually  and  those  who  worked  in  pairs. 

TABLE  1 

COMPARISON  OF  PAIR'S  AND  SINGLE'S 
ACHIEVEMENT  TESTS 


6R00P 

PAIRS 

SINGLES 


NUMBER 

OF  CASES  MEANS 


STANDARD 

DEVIATIONS 


57  14.2982  4.136 


29  13.3793 


4.981 


POOLED  VARIANCE  ESTIMATE 


T 

VALUE 


DEGREES  OF 
FREEDOM 


2-TAIL 

PROB. 


0.91 


84 


0.366 


Both  groups'  means  differed  by  less  than  one  question. 
The  pooled  variance  estimate  revealed  a  T  value  of  0.91  and 
a  2-tail  probability  of  0.366. 
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After  conducting  the  one-way  analysis  of  variance 
(ANOVA)  (see  Table  2)  it  was  obvious  that  a  significant 
difference  existed  somewhere  between  the  three  groups.  The 
means  of  the  three  groups  differed  significantly.  Although 
the  first  groups'  mean  of  15.2632  was  not  significantly 
different  from  the  second  groups'  mean  of  15.7222,  the  third 
group,  however,  had  a  much  lower  mean  than  either  of  the  two 
other  groups  of  12.1000. 

TABLE  2 

ONE-WAY  ANALYSIS  OF  VARIANCE 
WITH  MATCHED  PAIRS 


GROOP 

NUMBER 
OF  CASES 

MEANS 

STANDARD 

DEVIATIONS 

FI  W/  FI 

19 

15.2632 

4.1075 

FD  W/  FI 

18 

15.7222 

4.1559 

FD  W/  FD 

20 

12.1000 

3.3230 

DF 

SOM  OF 
SQUARES 

MEAN 

SQUARES 

F  F 

RATIO  PROB 

BETWEEN 

GROUPS 

2 

150.8345 

75.4173 

5.0459  .0098 

WITHIN 

GRODPS 

54 

807.0953 

14.9462 

TOTAL 

56 

957.9298 

A  Scheffe  Procedure  was  performed  to  support  the 
results  found  from  the  one-way  ANOVA.  Table  3  shows  that 
groups  made  up  of  either  two  FI  students  or  a  mixed  group  of 
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one  FD  and  one  FI  student  significantly  out-performed  groups 
made  up  of  two  FDs. 

TABLE  3 

SCHEFFE  PR(X:EDDRE 

GROUP  TYPE 

F  F  F 

I  D  D 

W/  W/  W/ 

F  F  F 

I  I  D 

MEAN  GROUP  TYPE 

15.2632  FI  W/  FI  * 

15.7222  FD  W/  FI  * 

12.1000  FD  W/  FD 

*  Denotes  Pairs  of  Groups  Significantly  Different 
at  the  0.050  Level 

Table  3  shows,  therefore,  that  FDs  benefit 
significantly  when  paired  with  a  FI.  FI  students  did 
equally  well  regardless  of  their  partner. 

According  to  the  literature,  FIs  are  analytical, 
independent,  do  not  seem  to  need  an  externally  provided 
structure  and  function  with  very  little  environmental 
support.  FDs,  on  the  other  hand,  show  a  lack  of  initiative 
and  have  a  readiness  to  submit  to  authority  (Witkin,  Lewis, 
Hertzman,  Machover,  Bretnall  Meissner  &  Wapner,  1954). 

330 


Table  4  shows  that  FD  students  paired  with  FI  students  or  FD 
students  paired  with  FD  students  enjoyed  working  in  pairs 
more  than  FI  students  paired  with  FI  students.  This  is, 
therefore,  consistent  with  the  literature  as  FI  individuals 
not  only  are  insensitive  to  social  cues,  but  also  function 
with  very  little  emotional  support.  FDs,  on  the  other  hand, 
are  more  oriented  towards  people  (Witkin,  1979). 

TABLE  4 

PAIRED  STUDENT  QUESTIONNAIRE  RESPONSES 
INCLUDES  MEAN,  SD,  AND 
PERCENTAGE  OF  REPONSES  PER  ITEM 

To  what  extent  did  you  enjoy  working  in  pairs? 

FIELD  INDEPENDENT/FIELD  INDEPENDENT 


VERY 

LITTLE 

1 

2 

3 

4 

5 

VERY 

IITTr*TI 

11% 

21% 

47% 

11% 

11% 

nuwxi 

MEAN  = 

2.895 

SD  = 

1.100 

FIELD  DEPENDENT/FIELD  INDEPENDENT 

VERY 

LITTLE 

1 

2 

3 

4 

5 

VERY 

MUCH 

12% 

12% 

29% 

24% 

24% 

MEAN  =  3.353  SD  =  1.320 
FIELD  DEPENDENT/FIELD  DEPENDENT 


VERY 

LITTLE 

1 

2 

3 

4 

5 

VERY 

MUCH 

10% 

15% 

10% 

15% 

50% 

MEAN  = 

3.800 

SD  = 

1.473 
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FD  students  paired  with  FD  students  helped  each  other 
more  during  training  than  any  other  group  (see  Table  5). 

This  is  also  consistent  with  the  literature  as  FDs  are  more 
helpful  than  FIs  (Witkin,  1979).  The  lower  mean  for  FDs 
matched  with  FIs  could  be  explained  by  the  fact  that  FIs  in 
this  group  were  not  interested  in  help,  even  if  provided. 
Although  FDs  matched  with  FDs  felt  that  they  had  been  helped 
more  during  the  lesson,  their  scores  were  th?'  lowest  of  the 
three  groups  (see  Table  2).  It  is  possible  that  this  group 
was  merely  socializing  and  not  exchanging  pertinent  class 
information. 
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TABLE  5 


PAIRED  STODENT  QUESTIONNAIRE  RESPONSES 
INCLUDES  MEAN,  SD,  AND 
PERCENTAGE  OF  REPONSES  PER  ITEM 


To  what  extent  did  your  partner  help  you  during  the  lesson? 


FIELD  INDEPENDENT/FIELD  INDEPENDENT 


VERY  1  2  3  4  5  VERY 

LITTLE  -  MUCH 

21%  16%  37%  16%  11% 

MEAN  =  2.895  SD  =  1.100 


FIELD  DEPENDENT/FIELD  INDEPENDENT 


VERY 

T  TmmT  r? 

1 

2 

3 

4 

5 

VERY 

uripo 

Ul  l  XULd 

18% 

18% 

35% 

24% 

6% 

MEAN  =  2.824  SD  =  1.185 
FIELD  DEPENDENT/FIELD  DEPENDENT 


VERY  1  2  3  4  5  VERY 

LITTLE  -  MUCH 

10%  10%  30%  35%  15% 


MEAN  =  3.350  SD  =  1.182 

The  next  two  tables  support  very  similar  areas  in  the 

literature.  FIs  not  only  learn  more  "efficiently”  in 

conditions  of  self  motivation,  but  also  do  not  seem  to  need 

an  externally  provided  structure  (Rosenberg,  Mints  &  Clark, 

1977).  FIs  also  tend  to  keep  to  themselves  or  desire  to 
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work  alone  (Witkin  &  Goodenough,  1981).  Table  6  shows  that 
almost  half  of  the  group  of  Pis  matched  with  FIs  desired  to 
work  alone.  Table  7  shows  that  90  percent  of  this  group 
believes  that  they  learn  more  by  doing  than  by  watching. 
These  results  are  in  contrast  to  other  groups  which  contain 
FD  participants.  FDs  prefer  to  work  physically  closer  to 
others  and  usually  perform  a  spectator  or  passive  learning 
role  (Witkin  &  Goodenough,  1977).  Table  6  shows  that  fewer 
FDs,  regardless  of  group,  prefer  to  work  alone.  Results  of 
Table  7  show  that  more  FDs,  in  general,  believe  they  learn 
more  by  watching  than  their  FI  counterparts. 
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TABLE  6 


PAIRED  STUDENT  QUESTIONNAIRE  RESPONSES 


INCLUDES  MEAN,  SD, 

r  AND 

PERCENTAGE  OF  REPONSES 

PER  ITEM 

I  prefer  to  work 

• 

FIELD  INDEPENDENT/FIELD 

INDEPENDENT 

LARGE 

SMALL 

ALONE 

GROUP 

GROUP 

1 

2 

3 

0% 

53% 

47% 

MEAN  =  2.474  SD 

=  .513 

FIELD  DEPENDENT/FIELD  INDEPENDENT 

LARGE 

SMALL 

ALONE 

GROUP 

GROUP 

1 

2 

3 

0% 

71% 

29% 

MEAN  =  2.294  SD 

=  .470 

FIELD  DEPENDENT/FIELD 

DEPENDENT 

LARGE 

SMALL 

ALONE 

GROUP 

GROUP 

1 

2 

3 

5% 

63% 

32% 

MEAN  =  2.263  SD  =  .562 
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TABLE  7 


PAIRED  STUDENT  QUESTIONNAIRE  RESPONSES 
INCLUDES  MEAN,  SD,  AND 
PERCENTAGE  OF  REPONSES  PER  ITEM 

I  learn  more  by  _ , 

FIELD  INDEPENDENT/FIELD  INDEPENDENT 

WATCHING  DOING 


1  2 


10% 

90% 

MEAN  =  1.895 

SD  =  .315 

FIELD  DEPENDENT/FIELD  INDEPENDENT 

WATCHING 

DOING 

1 

2 

18% 

82% 

MEAN  =  1.882 

SD  =  .485 

FIELD  DEPENDENT/FIELD  DEPENDENT 

WATCHING 

DOING 

1 

2 

21% 

79% 

MEAN  =  1.789 

SD  =  .419 
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DISCUSSION 


This  study  attempted  to  determine  wh“*-hct 
paired/cooperative  computer-assisted  instruction  (CAI)  is  as 
effective  as  an  individualistic  approach.  The  study  also 
examined  the  interactive  effects  of  individual  cognitive 
style  on  paired/cooperative  CAI.  Results  showed  that  no 
significant  difference  existed  between  the  mean  posttest 
scores  of  participants  who  worked  individually  and  those  who 
worked  in  pairs.  The  manner  in  which  individuals  are  paired 
by  individual  cognitive  style  also  made  a  significant 
difference  in  individual  achievement  test  scores. 

Groups  made  up  of  either  two  field  independent  students 
or  a  mixed  group  of  one  field  dependent  and  one  field 
independent  student  significantly  outperformed  groups  made 
up  of  two  field  dependents.  Although  field  dependents 
matched  with  field  dependents  enjoyed  working  in  pairs  more 
than  any  other  group,  the  optimal  pairing  included  one  field 
independent  student.  Field  independent  students  performed 
equally  well  regardless  of  their  partner  and  field  dependent 
students  benefited  significantly  when  paired  with  a  field 
independent  student.  Training  performed  in  this  manner  was, 
therefore,  found  to  be  not  only  more  efficient,  but  also 
more  effective. 
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ABSTRACT 


Interactive  computer-based  instructional  (CBI)  systems 
suffer  a  serious  deficiency:  limited  interactivity. 
Research  suggests  that  the  level  of  interactivity  is  an 
important  factor  in  the  effectiveness  of  CBI.  The 
technology  to  make  CBI  delivery  more  conversational  now 
exists.  Given  the  possibilities  of  more  conversational 
CBI  delivery,  there  is  a  need  to  establish  principles 
to  guide  the  use  of  this  technology.  The  Air  Force 
Human  Resources  Laboratory  is  conducting  a  series  of 
experiments  at  the  School  of  Aerospace  Medicine  to 
establish  prescriptions  for  the  use  of  auditory 
presentations  in  CBI.  This  paper  reviews  the  research 
findings  to  date  in  the  area  of  auditory  presentations 
in  CBI.  In  addition,  we  present  the  design  of  our 
initial  experiment  which  is  intended  to  test  the 
hypothesis  that  auditory  reinforcement  could  extend  the 
normal  memory  limitations  that  pertain  to  procedural 
knowledge. 


INTRODUCTION 

Recent  advances  in  computer 
technology  make  it  possible  to 
incorporate  speech  generation 
and  recognition  into  the 
construction  of  human-computer 
interfaces  (HCIs) .  Research 
in  cognitive  and  experimental 
psychology  indicates  that 
presentation  modality  has  an 
effect  on  recall  and  retention 
(cf.,  Gathercole  and  Conway, 
1988,  and  Penny,  1989) . 

Studies  of  computer-based 
instruction  (CBI)  tend  to 
indicate  that  the  level  of 
interactivity  influences  the 
effectiveness  of  the  instruc¬ 


tion  (Jonassen,  1985) .  Given 
the  current  trend  toward  more 
CBI  and  the  desire  to  make  CBI 
more  effective,  it  is  natural 
to  expect  that  auditory  pre¬ 
sentations  will  become  more 
common  as  CBI  becomes  more 
conversational  and  more  inter¬ 
active.  As  a  result,  there  is 
a  need  to  develop  principles 
for  the  optimal  use  of  audito¬ 
ry  presentations  in  CBI. 

This  paper  reviews  what  is 
known  about  computer-based 
auditory  presentations.  The 
research  literature  in  human 
factors  design,  experimental 
and  cognitive  psychology,  and 
educational  technology  is 
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reviewed  as  a  precursor  to 
establishing  prescriptions  for 
auditory  reinforcement  in  CBI. 
In  addition,  a  method  to  em¬ 
pirically  test  and  refine 
prescriptions  and  principles 
pertaining  to  auditory  presen¬ 
tations  is  discussed.  The 
first  experiment  in  an  experi¬ 
mental  suite  aimed  at  estab¬ 
lishing  guidelines  for  the  use 
of  auditory  presentations  is 
also  presented. 


REVIEW  OF  RELEVANT  RESEARCH 

A  growing  body  of  literature 
in  human  factors,  computer 
technology,  and  cognitive 
psychology  has  called  for  the 
Implementation  of  a  more 
versatile  and  meaningful  HCI 
(Cooper,  1987) .  The  trend  is 
toward  a  natural  communication 
link  between  user  and 
computer.  The  result  will  be 
more  conversational  computer 
systems.  The  potential 
benefit  to  CBI  is  that  the 
level  of  interactivity  can  be 
more  varied  and,  therefore, 
more  effective. 

Although  speech  is  arguably 
the  most  natural  form  of  in¬ 
terpersonal  communication,  it 
is  also  the  least  developed 
form  of  human-computer  commu¬ 
nication.  One  reason  for 
this,  of  course,  is  that  rea¬ 
sonable  digitization  and  com¬ 
pression  algorithms  are  rela¬ 
tively  new  and  that  only  re¬ 
cently  has  cost  ceased  to  be  a 
major  factor  in  adding  speech 
generation  and  recognition  to 
a  personal  computer.  The 
experiment  discussed  below 
makes  use  of  a  COVOX  speech 
generation  system.  The  COVOX 


system  is  also  capable  of 
voice  recognition  (all  for 
less  than  $200) ,  but  this 
aspect  of  HCI  was  not  explored 
in  this  experiment  due  to  the 
inherent  complexities  of  voice 
recognition. 

In  spite  of  the  widespread  use 
of  auditory  presentations  in 
learning  situations  from  pre¬ 
school  to  the  post-graduate 
level,  the  psychological, 
educational,  and  human-factors 
literatures  do  not  contain 
much  research  concerning  how 
humans  acquire  knowledge  and 
make  inferences  based  on  au¬ 
rally  presented  information. 
Not  much  is  known  about  how 
individuals  integrate  aural 
feedback  with  nonredundant 
information  and  what  skill 
determinants  in  this  area 
exist. 

The  guidelines  which  do  exist 
pertaining  to  auditory  presen¬ 
tations  are  mostly  intuitive 
or  commonsensical  and  pertain 
primarily  to  non-speech  audio 
cues.  The  table  below  depicts 
a  typical  example  of  a  deci¬ 
sion  procedure  for  determining 
whether  to  use  an  audio  or 
visual  channel  for  the  presen¬ 
tation  of  Information  (Deathr- 
idge,  1972); 


Use  auditory  display  if; 

The  message  is  simple. 

The  message  is  short. 

The  message  will  not  not 
be  referred  to  later. 

The  message  deals  with 
events  in  time. 
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The  message  calls  for 
immediate  action. 

The  visual  channel  is 
already  overloaded. 

The  receiving  location 
is  not  well  lit. 

The  person's  task  re¬ 
quires  continual  movement. 


Use  visual  display  if; 

The  message  is  complex. 

The  message  is  long. 

The  message  will  be  re¬ 
ferred  to  later. 

The  message  deals  with 
events  in  space. 

The  message  does  not  call 
for  immediate  action. 

The  audio  channel  is 
already  overloaded. 

The  receiving  location  is 
noisy. 

The  person's  task  does 
not  require  motion. 


Many  of  the  principles  sug¬ 
gested  in  this  adaptation  of 
Deathridge's  decision  proce¬ 
dure  have  not  been  empirically 
justified.  Indeed,  findings 
in  cognitive  science  suggest 
that  some  may  be  incorrect. 

For  example,  events  in  time 
can  be  given  a  spatial  repre¬ 
sentation.  The  essential 
consideration  is  determining 
whether  a  spatial  representa¬ 
tion  of  events  contributes  to 


a  meaningful  mental  model 
appropriate  to  the  learning 
task  at  hand. 

More  recent  research  reflects 
the  same  emphasis  on  non¬ 
speech  auditory  displays  and 
the  same  lack  of  rigorous 
empirical  justification.  For 
example,  Sanders  &  McCormick 
(1987)  propose  that  auditory 
displays  are  preferable  to 
visual  displays  when  one  or 
more  of  the  following  apply: 

1.  The  origin  of  the 
signal  is  itself  a  sound. 

2 .  When  continuously 
changing  information  of 
some  type  is  presented 
(see  also  Gaver,  1989) . 

3.  When  speech  channels 
are  fully  employed. 

4.  When  a  verbal  re¬ 
sponse  is  required. 

There  is  scant  research  about 
how  best  to  convey  different 
substantive  categories  of 
information  using  alternative 
auditory  displays.  For  exam¬ 
ple,  to  convey  information 
about  a  database,  a  new  use 
for  auditory  displays,  one 
must  convey  information  about 
appearances ,  states ,  struc¬ 
tures,  functions,  processes, 
etc.  (Sumlkawa,  1985;  Bly, 
1985;  Gaver,  1989).  To  do 
this  successfully  requires  a 
principled  means  of  choosing 
one  or  another  method  of  pre¬ 
sentation  as  being  best  suited 
to  a  particular  application. 

The  literature  reviewed  suf¬ 
fers  several  limitations. 
First,  the  main  emphasis  in 
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the  psychological  literature 
concerns  how  auditory  informa¬ 
tion  is  perceived  and  remem¬ 
bered.  There  is  little  psy¬ 
chological  research  on  how 
displays  can  be  used  to  best 
represent  particular  informa¬ 
tion  about  a  particular  sub¬ 
ject  domain  for  a  particular 
instructional  purpose  to  a 
particular  kind  of  student. 

A  second  limitation  in  the 
psychological  literature  is 
that  there  has  been  signifi¬ 
cant  emphasis  on  the  differ¬ 
ences  between  novices  and 
experts,  but  relatively  little 
emphasis  on  how  expertise  is 
acquired.  As  a  consequence, 
the  educational  implications 
of  much  of  the  psychological 
research  are  quite  limited. 

However,  the  psychological 
literature  does  suggest  that 
multiple  sensory  sources  have 
a  generally  positive  effect  on 
memory  (Wickens,  1984) .  In 
Wickens'  model,  separate  re¬ 
source  pools  are  hypothesized 
to  exist  for  different  types 
of  stimulus  codes  and  for 
different  stages  of  process¬ 
ing.  The  implication  for 
instructional  technology  is 
the  suggestion  of  these  audi¬ 
tory  principles: 

1.  Learning  is  facili¬ 
tated  when  stimuli  are 
input  to  different  chan¬ 
nels. 

2.  Learning  is  facili¬ 
tated  when  stimuli  are  of 
qualitatively  different 
types,  resulting  in  mul¬ 
tiple  encoding  of  input 
stimuli. 


The  costs  and  benefits  of 
multiple  resources  (presenta¬ 
tion  modalities)  and  redundan¬ 
cy  is  a  primary  focus  of  the 
research  described  below. 

In  general,  however,  the  re¬ 
search  emphasis  has  typically 
been  on  aural  versus  visual  or 
textual  presentation.  Rela¬ 
tively  little  consideration 
has  been  given  to  comparative 
evaluations  of  auditory  pre¬ 
sentations  in  various  instruc¬ 
tional  modalities  (e.g.,  feed¬ 
back,  explanation,  etc.). 

The  human  factors  literature 
emphasizes  the  principal  ergo¬ 
nomic  implications  of  using 
particular  delivery  media  and 
channels.  The  available  lit¬ 
erature  deals  with  signal 
intensity,  signal-to-noise 
ratios,  discriminabililty,  and 
the  like  (Smith  &  Goodwin, 
1970) .  The  following  existing 
principles  of  human  factors 
engineering  will  probably 
prove  to  be  extensible  to  the 
auditory  domain: 

1.  Selection  of  signal 
dimensions  and  encoding 
should  align  with  learned 
or  natural  relationships 
of  the  user  (e.g.,  using 
high  frequencies  to  sig¬ 
nify  an  increasing  val¬ 
ue)  . 

2.  When  presenting  com¬ 
plex  information  a  two 
stage  signal  should  be 
used  (e.g.,  use  one  sig¬ 
nal  to  attract  attention 
and  a  second  signal  to 
designate  particular 
information)  . 

3.  Stimuli  should  be 
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discernible  from  the 
environment. 

Other  similar  principles  can 
be  extracted  from  the  human 
factors  literature.  As  with 
the  psychology  and  educational 
literatures,  there  is  not  a 
lot  of  specific  research  per¬ 
taining  to  the  use  of  auditory 
presentations  in  CBI.  In 
addition,  many  human  factors 
principles  are  simply  intu¬ 
itive  and  not  empirically 
justified.  Much  research 
remains  to  be  accomplished 
before  we  fully  understand  the 
limits  of  the  human  auditory 
channel.  The  details  of  audi¬ 
tory  chunking,  auditory  over¬ 
load,  auditory  transfer,  and 
the  other  auditory  consider¬ 
ations  which  have  implications 
for  learning  and  the  design  of 
instruction  remain  to  be  dis¬ 
covered  . 


INITIAL  EXPERIMENTAL  DESIGN 

The  Human  Resources 
Directorate  (AL/HRTC)  and  the 
School  of  Aerospace  Medicine 
(SAM)  have  established  a  Memo¬ 
randum  of  Understanding  to 
allow  collaboration  on  pro¬ 
jects  of  mutual  interest  or 
benefit.  AL/HRTC  will 
conduct  a  series  of 
experiments  at  the  computer 
learning  facility  at  SAM  to 
establish  auditory 
prescriptions  to  guide  the 
optimization  of  auditory  pre¬ 
sentations  in  CBI. 

The  initial  experiment  is 
designed  to  test  the  hypothe¬ 
sis  that  auditory  reinforce¬ 
ment  can  improve  recall  and 
retention  of  procedural  knowl¬ 


edge.  A  COVOX  speech  system 
in  a  Z-248  microcomputer  pro¬ 
vides  the  speech  capabilities. 
A  lesson  module  authored  in 
TenCore  provides  the  CBI  set¬ 
ting.  Calls  to  the  speech 
system  are  made  via  the  Ten- 
Core  EXEC  command.  Digitized 
recordings  of  actual  human 
speech  will  be  used,  as  re¬ 
search  indicates  this  is  more 
likely  to  be  effective  than 
synthesized  speech  (Pisoni, 
1982) . 

The  lesson  module  involves  the 
completion  of  a  medical  form 
used  to  record  the  results  of 
several  types  of  medical  exam¬ 
inations.  This  material  is 
part  of  the  Aeromedical  Tech¬ 
nician  Course  taught  at  SAM. 
The  learning  tasks  are  primar¬ 
ily  procedural  in  nature.  The 
lesson  module  will  contain 
approximately  45  minutes  of 
student  CBI. 

Experimental  subjects  will  be 
familiarized  with  the  elements 
of  the  CBI  system,  including 
the  keyboard,  the  screen,  and 
the  speech  system,  prior  to 
the  initiation  of  the  experi¬ 
ment.  During  this  familiar¬ 
ization  phase,  subjects  will 
be  queried  or  presented  infor¬ 
mation  using  auditory,  graphi¬ 
cal,  and  textual  presenta¬ 
tions. 

The  main  body  of  the  lesson 
module  will  be  presented  using 
alternating  blocks  of  nontest- 
able  and  testable  material. 
This  approach  allows  student 
motivation  to  be  determined  at 
various  points  in  the  instruc¬ 
tional  sequence  (Lepper  & 
Malone,  1985). 
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Six  instructional  treatments 
will  be  examined;  1)  audito¬ 
ry,  2)  textual,  3)  combined 
auditory  and  textual,  4)  com¬ 
bined  auditory  and  graphical, 
5)  combined  graphical  and 
textual,  and  6)  combined  audi¬ 
tory,  textual,  and  graphical. 
The  point  of  selecting  these 
presentation  combinations  is 
to  explore  the  multiple  re¬ 
source  hypotheses  elaborated 
earlier. 


MEASUREMENT  AND  ASSESSMENT 

Delayed  retention  will  be 
measured  in  terms  of  recall. 
Subjects  will  be  administered 
constructed  response  items  to 
determine  the  extent  of  their 
retention.  Recognition  is  not 
measured  in  this  study,  be¬ 
cause  the  medical  form  cannot 
be  legitimately  used  by  some¬ 
one  who  is  only  capable  of 
recognizing  the  correct  proce¬ 
dures  and  lacks  the  ability  to 
perform  those  procedures. 

Measurement  and  testing  condi¬ 
tions  of  this  study  will  re¬ 
semble  as  closely  as  possible 
those  currently  in  use  in  the 
existing  Aeromedical  Techni¬ 
cian  course.  Therefore  the 
criterion  for  delayed  recall 
will  require  that  the  subject 
demonstrate  the  correct  proce¬ 
dure  for  completing  a  particu¬ 
lar  item  on  the  medical  form. 
All  subjects  will  be  tested 
after  a  delay  of  one  day,  as 
is  the  case  in  the  existing 
Aeromedial  Technician  course. 
An  added  benefit  of  keeping 
the  measurement  and  testing 
conditions  as  close  currently 
in  use  is  that  comparisons 
with  existing  databases  can  be 


made  to  determine  the  general 
efficacy  of  the  CBI  course 
module  relative  to  the  class¬ 
room-based  module. 

As  an  additional  measure  of 
the  impact  of  the  different 
instructional  presentation 
methods  in  the  absence  of  any 
opportunities  for  rehearsal , 
some  of  the  ostensibly  "non- 
testable”  material  will  be 
tested.  Thus,  it  will  be 
possible  to  assess  the  effects 
of  the  instructional  presenta¬ 
tions  without  the  likelihood 
of  inflated  per  ormance  due  to 
rehearsal  during  the  instruc¬ 
tion. 


CONCLUSION 

The  implementation  of  auditory 
presentations  should  be 
systematically  planned  and 
designed  with  the  entire 
instructional  system  in  mind. 
Matching  media,  types  of 
instructional  ob j  ectives , 
types  of  subject  matter,  and 
learning  characteristics  is 
not  a  simple  task.  Guidelines 
for  the  design  of  interactive 
courseware  should  include 
prescriptions  for  the  use  of 
auditory  displays  that  maxi¬ 
mize  the  efficiency  of  the 
auditory  channel  for  particu¬ 
lar  types  of  material  and 
learning  objectives. 

The  Advanced  Instructional 
Design  Advisor  (AIDA)  project 
at  AFHRL  is  an  attempt  to 
provide  courseware  authoring 
guidelines  to  courseware  de¬ 
velopers  (Muraida  &  Spector, 
1990) .  Clearly,  AIDA  will 
need  the  capability  to  pre¬ 
scribe  when  auditory  presenta- 
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tions  are  preferable  to  textu¬ 
al  or  graphical  presentations, 
to  suggest  what  sorts  of  audi¬ 
tory  presentations  are  most 
likely  to  be  effective  for  a 
particular  purpose,  and  to 
indicate  what  auditory  conven¬ 
tions  should  be  observed. 

As  learning  technology 
evolves,  there  will  be  an 
increasing  role  for  the  use  of 
advanced  technologies  such  as 
speech  recognition  and  artifi¬ 
cial  neural  networks  in  inter¬ 
active  courseware  (Spector, 
1990) .  In  short,  the  demand 
for  principles  and  guidelines 
for  the  effective  use  of  audi¬ 
tory  presentations  in  CBI  will 
continue  to  grow.  As  a  conse¬ 
quence,  conducting  experiments 
to  establish  these  principles 
will  play  a  vital  role  in  the 
success  of  future  learning 
systems . 
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COMPUTER-BASED  VOICE  RECOGNITION  TECHNOLOGY 
IN  FUNCTIONAL  FOREIGN  LANGUAGE  TRAINING 


Abstrac± 

The  Futures  Training  Division  of  the  U.S.  Army  Training 
and  Doctrine  Command  is  demonstrating  the  application  of  a  new 
technology  that  eliminates  any  self  certification  of  language 
pronunciation  on  the  part  of  the  foreign  language  student.  The 
computer  technology  incorporates  speaker  independent  speech 
recognition  that  evaluates  discrete  speech  (words  or  phrases 
spoken  within  4  seconds)  on  accuracy  of  pronunciation  and 
enunciation. 

This  automated  system  is  used  to  replace  the  tape  player 
used  for  audio  recording  and  playback.  It  provides  a  drill  and 
practice  capability  that  is  interactive,  and  thus,  the  student 
becomes  completely  involved  in  all  aspects  of  learning  to  speak 
that  language.  The  system  teaches  the  basic  skills  — vocabulary, 
pronunciation,  and  syntax —  required  to  learn  a  foreign 
language.  Once  the  student  has  acquired  this  skill, 

conversational  experience  may  be  obtained  in  either  a  classroom 
or  a  natural  environment. 

Computer  based  instruction  drill  and  practice  lessons, 
utilizing  voice  recognition  technology,  have  been  developed  in 
Spanish  and  French  for  functional  language  training  of  the  Army's 
Special  Forces  units.  In  an  on-going  study,  the  lessons  are 
being  evaluated  to  determine  their  training  and  cost 
effectiveness  as  compared  to  traditional  language  lessons. 
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COMPUTER  BASED  VOICE 
RECOGNITION  TECHNOLOGY  IN 
FUNCTIONAL  FOREIGN  LANGUAGE 
TRAINING 

Marta  J.  Bailey  and 
Gary  G.  Wright 


Instructional  methodology 
in  the  area  of  foreign 
language  training  is  a 
widely  discussed  topic  among 
educators.  There  are  groups 
of  educators  who  prefer  a 
complete  immersion  of  the 
student  in  the  target 
language  and  a  group  of 
educators  who  prefer  the 
more  traditional  method  of 
teaching  strictly  defined 
grammar  rules  that  emphasize 
the  written  capabilities  of 
the  new  speaker.  One  method 
is  unlikely  to  be  agreed 
upon  by  all  educators. 

A  common  element  of  both 
methods  is  the  time  the 
student  must  devote  to 
practicing  pronunciation  and 
learning  new  vocabularies. 
The  most  traditional  method 
involves  spending  time  in  a 
language  laboratory  wearing 
a  headset  in  order  to  listen 
to  a  native  speaker  and 
practicing  speaking  the 
language  by  recording  your 
voice  and  listening  tr  the 
playback.  This  activity  is 
generally  a  most  boring  task 
and  it  is  extremely 
difficult  to  apply  one's 
total  concentration. 

Certification  of  when  and 
how  well  you  are  speaking 
the  target  language  is 
strictly  a  judgement  call 
made  by  the  speaker.  Self 
certification  may  be 

accomplished  in  a  relatively 
short  amount  of  time. 


This  paper  will  describe  the 
application  of  a  new 
technology  that  eliminates 
any  self  certification  of 
language  pronunciation  on 
the  part  of  the  student. 
The  computer  technology 
incorporates  speaker 

independent  speech 

recognition  that  evaluates 
discrete  speech  (words  or 
phrases  i^poken  within  4 
seconds)  on  accuracy  of 

pronunciation  and 

enunciation.  This  automated 
system  replaces  tape  players 
used  for  audio  recording  and 
playback.  It  provides  a 
drill  and  practice 

capability  that  is 

interactive  and  thus  the 
student  becomes  completely 
involved  in  all  aspects  of 
learning  to  speak  the 
language.  The  system 

teaches  the  basic 

skill- -vocabulary  and 

pronunciation — required  to 
learn  a  foreign  language. 
Once  the  studen^  has 
acquired  this  skill, 

conversational  experience 
may  be  obtained  in  either  a 
classroom  or  a  natural 
environment. 

This  project  is  a  part  of 
the  Army's  concept  of 

distributed  training  whereby 
a  substantial  percentage  of 
training  will  be  delivered 
outside  of  the  traditional 
school  house  setting.  The 
Futures  Training  Division  of 
the  Army  Training  and 
Doctrine  Command  (TRADOC)  is 
sponsoring  a  pilot  research 
project  in  conjunction  with 
EER  Systems  and  the  John  F. 
Kennedy  Special  Warfare 
Fighting  Center  and  School 
at  Ft.  Bragg,  North 
Carolina. 


At  this  school,  sixteen 
different  languages  are 
taught  to  military  special 
forces  units  who  must 
communicate  with  both 

friendly  and  enemy  forces 
once  they  have  landed  on 
foreign  soil.  The 

methodology  involves 

teaching  the  language  in  a 
structured  functional 

approach.  There  are  ten 

basic  lessons  that  deal  with 
the  structure  of  the 
language  and  a  generic  set 
of  words  (numbers, 

salutations,  etc.) .  An 

additional  38  lessons 

involve  functional  areas 

such  as  medical  tasks, 

demolition,  communications 
and  so  on.  These  tasks 
require  a  vocabulary  of 

approximately  2500  words 

and/or  phrases. 

The  hardware  components  of 
the  language  training  system 
are  an  IBM  compatible 
personal  computer  such  as  a 
Zenith  248,  BIDS,  or  laptop 
and  an  off-the-shelf  signal 
processing  board.  To 

complete  the  delivery 

system,  speech  recognition 
software  and  mission 

oriented  training  scenarios 
are  developed.  The  most 
significant  feature  of  this 
language  training  system  is 
that  it  provides  an 
interactive  capability  for 

the  student  to; 

Hear  a  word  or  phrase 
pronounced  properly  by  a 
native  speaker,  and 


-  Sav  the  word  or  phrase 
back  to  the  system,  and 


Receive _ Immediate 


Feedback 

from 

the  system 

that  will 

tell 

the  student 

if  the  word  or 

phrase 

was 

pronounced 

correctly 

or 

incorrectly 

,  and 

Reoeat 

the 

words 

and 

phrases  again 

and  again 

until  the 

pronunciation 

is 

correct. 

The  system  is  very  close  to 
having  your  own  personal 
instructor,  because 

essentially  you  talk  to  the 
system  and  it  talks  back. 

The  most  important  aspect  of 
the  speech  recognizer  is  its 
ability  to  recognize  any 
person's  speech  regardless 
of  accent,  tone,  or 
variations  in  speech 

characteristics  associated 
with  the  sex  of  the  speaker. 

The  primary  objective  is  for 
the  speaker  to  pronounce  the 
word/phrase  as  it  has  been 
pretrained  by  native 

language  speakers.  Most 

recognizers  in  today's 

market  try  to  accommodate 
for  any  speaker;  this  system 
requires  the  speaker  to 
accommodate  to  the  correct 
pronunciation.  If  a 

misrecognition  or 

nonrecognition  occurs,  it  is 

a  measure  of  poor 
performance  (pronunciation) 
and  not  an  error  in  the 
recognition  system. 
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Another  important  feature  is 
the  capability  to  train  a 
specific  language  dialect. 
For  example,  if  the  mission 
requirement  necessitates 

proficiency  in  Colombian 
Spanish,  the  system  can  be 
"trained"  to  "speak  and 
understand"  Colombian 

Spanish.  If  the  requirement 
is  to  have  a  broad-based 
Spanish  capability,  then  the 
system  would  be  trained 
across  a  variety  of  Spanish 
speakers. 

The  system  can  train 
vocabulary  and  phrases  up  to 
four  seconds  in  length.  A 
four  second  phrase  on 
average,  translates  into  six 
to  eight  words,  which 
includes  the  majority  of 
military  requirements. 

The  system  is  easily 
transportable  which  makes 
standardized  sustainment 

training  at  home  station  a 
reality. 

The  application  is  designed 
to  teach  individual  words; 
providing  the  speaker  the 
opportunity  to  listen  to 
each  word  as  modeled  by  a 
native  speaker.  After  all 
words  have  been  pronounced 
correctly,  the  words  are 
grouped  into  phrases  until  a 
functional  sentence  has  been 
spoken.  If  the  sentence  is 
more  than  4  seconds  in 
length,  the  system  will  not 
recognize,  but  will  digitize 
the  speaker's  voice  and 
provide  playback  so  that  the 
speaker  may  listen  to 


his/her  pronunciation. 

The  present  method  of 
teaching  foreign  language  at 
the  school  involves  an 
intensive  8-10  hour  day  for 
each  student.  Soldiers  spend 
approximately  six  hours  in 
class  practicing  speaking. 
Each  evening,  they  are 

required  to  learn  a  new 
vocabulary  of  about  50 

words.  They  practice 

speaking  the  new  vocabulary 
and  then  complete  some  paper 
based  exercises.  They  use 
cassette  tape  players  to 
learn  the  new  vocabulary,  to 
model  the  pronunciation,  and 
to  receive  instructions  to 
complete  the  exercises.  The 
following  day,  they 

generally  spend  the  first 
hour  or  two  in  class  in 
learning  the  vocabulary  as  a 
group  which  suggests  that 
homework  practice  time  has 
not  been  as  effective  as 
desired.  Time  spent 

reviewing  homework  reduces 
the  conversational  practice 
to  approximately  four  hours. 

The  use  of  the  automated 
system  is  expected  to  reduce 
the  student's  learning 
time.  If  they  know  their 
vocabulary,  they  could  use 
the  first  4  hours  of  the  day 
to  practice  conversational 
exposure,  then  they  could 
use  the  afternoon  (in  a 
laboratory  environment)  to 
learn  and  practice  the  new 
vocabulary  for  the  next  day 
and  complete  more  meaningful 
exercises  administered  by 
the  computer. 
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The  systems  computer 

management  function  will 
also  provide  a  documented 
audit  trail  of  the  use  the 
system  to  see  how  much  time 
is  needed  to  learn  new 
vocabularies.  This 

information  could  be 

valuable  in  future 

development. 

TRADOC  provided  12  computers 
with  voice  recognition 

capability  for  a  learning 
lab  setting  at  the  JFK 
Special  Warfare  Fighting 
Center  and  School.  Ideally, 
all  students  should  receive 
a  personal  computer  for  home 
use  during  the  duration  of 
language  training.  Spanish 
and  French  drill  and 
practice  lessons  will  be 
formally  evaluated  for 
training  and  cost 

effectiveness. 


The  potential  value  of  this 
approach  to  language 

training  lies  primarily  in 
unit  based  sustainment 
training.  Language  training 
deteriorates  rapidly  without 
use  and  practice.  Both  the 
active  and  reserve  Army 
forces  have  strong  needs  for 
affordable,  convenient 

sustainment  training. 

The  hardware  costs  for  this 
system  are  nominal.  A 

$1500.00  signal  processing 
board  is  added  to  a  standard 
AT  compatible 

microprocessor.  Research 

and  development  costs  of 
less  than  one  million 
dollars  have  brought  the 
programmers  to  the  point  of 
being  capable  of  rapidly 
developing  additional 

languages. 
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THE  DEVELOPMENT  OF  ALTERNATIVE  STRATEGIES 


John  E.  Buckley 
Abstract 

Our  current  training  system  relies  on  three  training 
strategies,  one  for  officers,  one  for  warrant  officers,  and 
one  for  enlisted  personnel.  Training  is  descriptive  in 

nature.  All  soldiers  receive  the  same  training  or  type  of 
training  without  regard  to  the  skills  and  knowledges  of  the 
individual  or  the  complexity  of  the  training.  In  an  era  of 
declining  budgets,  this  may  not  be  the  most  cost-effective 
method  of  organizing  the  training  system. 

As  a  case  in  point,  fifty  percent  of  the  Army  are  in 
their  first  term  of  service.  During  this  period,  they  attend 
Basic  Training  and  Initial  Entry  Training.  This  accounts  for 
approximately  fifty  percent  of  our  training  dollars.  The 
return  on  these  training  dollars  is  questionable,  since  only 
thirty-three  percent  will  reenlist  for  a  second  tour. 
Training  options  must  address  either  increasing  the  number  of 
reenlistments  or  shifting  the  training  emphasis  to  focus  on 
those  who  plan  to  make  the  Army  a  career  not  a  short  term  job. 

To  assist  in  the  analysis  and  development  of  alternative 
strategies,  a  two  dimensional  model  was  constructed.  It  is 
predicted  on  the  assumption  that  different  clusters  of 
occupations  share  certain  characteristics  which  can  be  used  to 
custom  tailor  training  strategies. 
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ALTERNATIVE  STRATEGIES 
John  E.  Buckley 


The  Armies'  current  training 
system  relies  on  three 
training  strategies — 
one  for  officers,  one  for 
warrant  officers,  and  one 
for  enlisted  personnel. 
Training  is  descriptive  in 
nature.  All  soldiers 

receive  the  same  training  or 
type  of  training  based  on  a 
program  of  instruction 
structured  to  a  traditional 
service  school  environment. 
This  is  without  regard  to 
the  skills  and  knowledges  of 
the  individual  or  the 
complexity  or  nature  of  the 
training  required. 


The  Enlisted  Personnel 
Management  System  has 
classified  33  Career 

Management  Fields  (CMF). 

Each  is  further  broken  down 
into  clusters  of  Military 
Occupational  Specialties 

(MOS)  which  total  332. 
These  range  in  training 
complexity  from  simple 

(i.e..  Motor  Transport 

Operator)  to  complex  (i.e. 
Patriot  Operator  &  Systems 
Mechanics).  Training 

duration  in  the  service 
schools  varies,  accordingly, 
from  a  minimum  of  13  weeks 
to  a  maximum  of  46  weeks 
training.  There  is  also  a 
wide  divergence  in  the 
physical  and  psychological 
demands  associated  with  each 
MOS.  Some  are  primarily 
psychomotor,  others  are 
highly  cognitive.  Some  are 


highly  technical,  others 
highly  tactical.  In  looking 
at  the  total  spectrum  of 
enlisted  training,  it  became 
readily  apparent  that  the 
"one  size  fits  all"  training 
approach  may  not  be  the  most 
cost-effective.  The 

situation  is  further 

exacerbated  by  a  high 
turn-over  rate  in  many  MOS. 

As  a  case  in  point,  fifty 
percent  of  the  soldiers  are 
in  their  first  term  of 
service.  During  this 

period,  they  attend  Basic 
Training  and  Initial  Entry 
Training  which  accounts  for 
approximately  fifty  percent 
of  the  Armies'  training 
dollars.  The  return  on 

these  training  dollars  is 
questionable  since  Armywide 
reenlistment  rates  average 
only  thirty  three  percent. 
To  bring  the  dilemma  into 
sharper  focus,  in  1988  the 
Army  trained  5022  Light 
Wheeled  Vehicle  Mechanics. 
Of  these,  only  788 

reenlisted  as  Light  Wheeled 
Vehicle  Mechanics  and  1542 
reenlisted  for  other  MOS 
which  created  an  additional 
retraining  requirement.  The 
remaining  2692  personnel 
left  the  service  after  their 
first  enlistment.  An 

analysis  of  20  other  MOS 
showed  similar  patterns.  In 
order  to  provide  a  greater 
return  on  initial  entry 
training  dollars,  alterna¬ 
tive  strategies  should  be 
explored.  The  challenge  was 
to  develop  a  methodology  for 
clustering  MOS  so  that  these 
alternative  strategies  could 
be  focused  on  occupations 
sharing  similar  training 
characteristics. 
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•fo  assist  in  the  analysis 
and  development  of  these 
alternative  strategies  a  two 
dimensional  model  was 
constructed  (See  Figure  1). 
It  is  predicated  on  the 
assumption  that  different 
CMF  have  different  training 
complexities  which  warrant 
different  strategies,  and 
that  individual  differences 
in  soldiers  may  warrant 
different  approaches.  It 

also  assumes  that  the  Army 
may  not  be  able  to  sustain 
its  current  training  base  in 
the  future.  As  such,  it 
must  look  externally  to  meet 
some  of  its  future  training 
requirements. 

See  Figure  1. 

Tht  X-Axis  denotes  the 

tr?ining  complexity  of  the 
Cai.eer  Management  Field. 
The'  Y-Axis  denotes  how 

uniquely  military  the 

trrining  may  or  may  not  be. 
The  more  green,  the  more 
unique  the  training  is  to 
the  Army  environment. 

As  shown,  career  management 
ficLds  can  be  coarsely 

grt  uped  together  based  on 
their  greenness  or  their 
un  queness  to  the  Army,  and 
whether  the  technological 
training  requirements  are 
high  or  low.  Four 

classifications  of  CMFs 
result  from  this  clustering. 

A  further  analysis  indicates 
that  the  resulting  clusters 
share  certain  character¬ 
istics  which  may  warrant 
different  training 

strategies. 


In  classifying  CMF's  in  this 
manner,  it  is  important  to 
note  that  a  sophisticated 
highly  technical  weapon 
system,  like  the  Stinger, 
may  in  fact  have  a  low 
technological  training 

requirement.  (Some  experts 
have  amplified  either  real 
or  perceived  man-machine 
interface  problems  with  this 
system.  Although  approxi¬ 
mately  seventeen  steps  are 
required  to  fire  a  Stinger, 
they  are  relatively  simple 
sequential  steps  best 

learned  through  drill  and 
practice.) 

Although  it  has  been  useful 
to  look  at  enlisted  training 
at  a  macro- CMF  level,  it 
became  readily  apparent  as 
exceptions  surfaced,  that  in 
order  to  fine  tune  the 
strategies,  individual  MOS 
would  have  to  be  examined. 
For  example  as  shown  in 
Figure  2,  while  CMF  16 
generally  shares  the 

characteristics  associated 
with  dark  green,  low-tech 
MOS,  the  CMF  does  contain  an 
MOS  which  can  be  classified 
in  a  different  quadrant. 

See  Figure  2. 

CMF-16  Air  Defense  Artillery 

DARK  GREEN.  LOW  TECH 
QUADRANT 

These  Career 
Management 
Fields  contain 
MOS  which 

involve  the 
operation  and  maintenance  of 
military  peculiar  equipment 
or  systems.  As  such,  there 


are  no  existing  civilian 
institutions  which  train 
these  occupations.  Many  of 
the  Combat  Arms  MOS  fall 
into  this  quadrant. 

One  of  the  systemic  problems 
in  the  service  schools  is  a 
lack  of  tactical  equipment 
for  training.  In  order  to 
utilize  scarce  tactical 
equipment  to  its  greatest 
advantage,  functional 

context  training  in  the  unit 
may  have  merit.  All 

soldiers  would  receive  an 
abbreviated  common/ 
generic  AIT  in  conjunction 
with  basic  training  after 
which  they  would  report  to  a 
unit  for  supervised 

on-the-job  training  (SOJT) 
enhanced  by  Distributed 
Training.  This  strategy 

gets  the  soldier  to  the  unit 
sooner  and  saves  a  Permanent 
Change  of  Station  (PCS)  move 
which  results  in  substantial 
savings. 

Since  these  are  the  most 
inexpensive  MOS  to  train, 
(from  an  individual  soldiers 
perspective)  a  revamping  of 
our  recruitment  efforts  may 
be  warranted.  Personnel 

could  be  recruited  into  dark 
green,  low  tech  CMF's  and  be 
offered  more  sophisticated, 
costly  training  as  an 
incentive  to  reenlist. 

Implementation  of  such  a 
policy  would  provide  more 
return  on  the  training 
investment  by  providing  more 
expensive  training  to  the 
thirty  three  percent  of  the 
population  who  demonstrate 
an  interest  in  remaining  in 
the  service  by  reenlisting. 
(Further  advance  training 


could  be  provided  to  the 
seventeen  percent  of  this 
population  who  reenlist  for 
a  third  tour.) 

In  summary,  the  dark  green, 
low  technology  MOS  share  the 
following: 

CHARACTERISTICS: 

1.  Contains  MOS  which  are 
not  readily  found  being 
trained  by  the  civilian 
sector  or  industry. 

2.  Are  of  relatively  short 
duration. 

3.  Have  low  technological 
training  demands. 

4.  Representative  MOS: 

a.  IIB  Infantryman 

b.  16S  Manportable  and 
Pedestal  Mounted  Stinger 
Crewmember 

c.  12B  Combat  Engineer 

5.  Have  relatively  low  Army 
Services  Vocational  Aptitude 
Battery  Requirements. 

6.  More  Psychomotor  than 
Cognitive. 

7.  More  tactical  than 
technical. 

8.  High  Density. 

9.  Require  ranges/ 
simulations  for  optimum 
training. 


STRATEGIES: 


1.  Focus  recruitment 
efforts  on  first  enlistment. 

2.  Best  trained  in  a  unit 
environment. 

3.  Permanent  Change  of 
Station  to  Unit  after  a 
"generic  Advanced  Individual 
Training." 

4.  Train  in  COHORT  context. 

5.  PCS  to  Unit  for  SOJT. 


DARK  GREEN.  HIGH  TECH 
QUADRANT 


peculiar 
systems, 
involve 
highly 
systems 
degree 
electrical 
system 


This  quadrant, 
like  the  pre¬ 
vious  one,  is 
characterized 

by  military 
hardware  or 

Most  of  the  CMFs 
the  maintenance  of 
complex  weapons 
requiring  a  high 
of  knowledge  of 
and  hydraulic 
operation  and 

Some  examples 
Patriot 


repair. 

include  the  Hawk  and 
Weapon  systems.  Training 
duration  exceeds  30  weeks 
for  most  of  the  MOS. 


From  a  population  density 
perspective,  it  appears 
feasible  that  an  MOS 
migration  path  could  be 
created  to  transition  second 
term  enlistments  from  the 
Low  Tech,  Dark  Green  CMFs. 
Personnel  who  served  as 
operators  for  2-3  years  on 
the  systems  would  have 
already  acquired  such 

substantial  backgrounds  in 
such  areas  as  operator 
preventative  maintenance. 


system  geometry,  nomencla¬ 
ture,  normal  and  abnormal 
system  operational  charact¬ 
eristics.  They  would, 

therefore,  require  sub¬ 
stantially  less  training 
then  current  maintenance 
personnel  are  receiving. 
Opportunities  through  the 
tuition  assistance  program 

could  provide  electronics 
training  via  community 
colleges  to  operators  who 
express  a  desire  to  reenlist 
into  the  high  tech,  dark 
green  CMFs.  If  this 

training  occurred  prior  to 
the  second  enlistment,  it 
would  reduce  training  in  the 
electronic  dark  green,  high 
tech  MOS,  by  approximately 
10  weeks.  This,  coupled 

with  first  enlistment  skills 
and  knowledges  training,  has 
the  potential  of  reducing 
training  for  these  long  MOS 
by  thirty  to  forty  percent. 

In  summary,  CMFs  in  this 
quadrant  share  the 

following: 

CHARACTERISTICS: 

1.  Contain  MOS  which  are 
not  readily  found  trained  by 
civilian  sector  or  industry. 

2.  Are  of  relatively  long 
duration. 

3.  May  have  enabling  skills 
and  knowledges  found  in 
civilian  sector. 

4.  Have  high  technological 
training  demands. 

5.  Tend  to  be  most  costly 
(Per  soldier). 

6.  More  Cognitive  than 
Psychomotor. 
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7.  Representative  MOS: 

a.  24T  Patriot  Operator 
Maintainer 

b.  27K  hawk  Firing 
Section  Repairer 

c.  33Q  E/W  Intercept 
Strategic  Processing/ 

Storage  Equipment  Repairer 

STRATEGIES; 


1.  Offer  as  an  incentive 
for  second  reenlistment. 

2.  Best  trained  in  service 
school  environment. 

3.  May  include  "front 
loaded"  enabling  skills  and 
knowledge  training. 

4.  Develop  career  migration 
patterns  to  transition  from 
other  similar  "Low  Tech"  MOS 
held  during  the  first 
enlistment. 

5.  "Grow"  Maintenance 
Personnel  from  Operator  "Low 
Tech,  Dark  Green  CMFs. 

LIGHT  GREEN.  LOW  TECH 
QUADRANT 

The  tasks 
associated  with 
the  MOS  in  this 
quadrant  are 

readily  found 
in  the  civilian  sector. 
Generally,  these  are  the 
"blue  collar"  military 
occupational  specialties. 

Strategies  which  focus  on 
recruitment  by  audition  or 
achievement  will  help  reduce 
front-end  training  costs. 


This  will  require  the 
adoption  of  novel  more 
aggressive  recruitment 

practices.  For  example,  the 
Army's  total  requirements 
for  band  members  could  be 
met  by  proactive  recruit¬ 
ment.  If  recruiters  visited 
the  various  High  School 
Bands  in  the  nation  and 
toured  with  the  All  American 
High  School  Band,  they  could 
offer  a  career  in  music  to 
high  school  seniors.  In 

this  particular  field,  the 
Army  would  have  little 
competition  and  could  offer 
an  advanced  promotion  if  the 
enlistees  could  pass  a 
certification  test.  Since 

we  are  recruiting  personnel 
who  already  possess  the 
skills  we  need,  there  would 
be  no  need  to  send  them  to 
AIT.  They  could  proceed  to 
a  Unit  immediately  after 
BCT.  Ultimately,  this  would 
lead  to  a  closing  of  the 
Music  School  at  a  sub¬ 
stantial  savings.  While 

this  example  is  probably  the 
most  dramatic,  the  same 
strategy  could  be  applied  to 
the  other  low  tech,  light 
green  MOS,  especially  in 
those  areas  commonly  taught 
in  high  school  vocational 
education  programs. 

CHARACTERISTICS: 

1.  Subjects  being  taught  in 
civilian  sector. 

2.  Are  of  relatively  short 
duration. 

3.  Have  low  technological 
training  demands. 
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4.  Involve  simple  hardware 
or  tasks  found  in  civilian 
sector. 

5.  Relative  low  cost  to 
train  per  student. 

6.  Representative  MOS: 

a.  88M  Motor  Transport 
Operator 

b.  71L  Administrative 
Specialist 

c.  63  B  Light  Wheeled 
Vehicle  Mechanic 

7.  More  psychomotor  than 
cognitive. 

STRATEGIES: 

1.  Incorporate  civilian 

off-shelf  courseware. 

2.  Utilize  community 

colleges,  high  schools,  or 
industry  to  train. 

—  Train  in  high  school 
Military  Science  Vocational 
Program. 

Attend  community 
college  either  after  or 
prior  to  BCT. 

3.  Recruit  by  achievement 
or  audition  verses  ability. 

4.  Contract  for  training  or 
for  services  provided. 


LIGHT  GREEN.  HIGH  TECH 
QUADRANT 


are  generally  considered 
paraprofessional  or  pro¬ 
fessional  occupations  by  the 
civilian  sector.  They 

generally  have  lengthy 

formal  educational  require¬ 

ments  such  as  associate  or 
bachelors  degree  educational 
programs.  This  provides  the 
concentration  of  cognitive 

skills  in  areas  such  as 

mathematics,  science,  and 
english  required  by  many  of 
the  CMFs  in  this  quadrant. 
Many  of  the  occupations  also 
have  licensing  requirements 
and  national  standards  as  in 
the  case  health  services 
CMFs.  All  require 

sophisticated  cognitive 

skills  and  an  ability  to 
perform  abstract  reasoning. 

CHARACTERISTICS: 

1.  Subjects  taught  by 
civilian  sector. 

2.  Are  of  relatively  long 
duration. 

3.  Have  high  technological 
demands. 

4.  Involve  sophisticated 

hardware  or  require  high 
technical  knowledge. 

5.  Relative  high  cost  to 
train. 

6.  Representative  MOS: 

a.  91C  Practical  Nurse 

b.  93H  Air  Traffic 
Control  Radar  Controller 


The  CMF's  in 
this  quadrant 


c.  35K  Avionics 
Mechanic 
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STRATEGIES; 


existing 


1.  Recruit  by  achievement 
verses  ability. 

2.  Create  incentives  for 
recruitment  of  community 
college  graduate. 


1.  Utilize 

on-going  programs. 

ADVANTAGES: 

a.  Probably  cheapest 
approach. 


3.  Mold/ influence  technical 
training  in  high  school, 
community  colleges,  and 
industry. 

4.  Utilize  total  national 
training  resources. 

5.  Offer  as  incentives  for 
second  term  enlistment. 


r 
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LIGHT  GREEN  MOS 

One  basic  premises 
underlying  the 
identification  of 
the  "light  green" 
MOS  is  that  in 
this  era  of  declining 
resources,  the  Army  must 
capitalize  on  the  total 
training/educational  assets 
of  the  nation  to  meet  some 
of  its  training  needs. 
These  assets  include  public 
educational  and  training 
facilities  and  commercially 
available  off-the-shelf 

courseware  which  can  be  used 
to  traiii  the  179  "light 
green"  MOS.  Even  a  cursory 
analysis  shows  training  is 

occurring  in  the  public 

sector  that  mirrors  military 
service  schools.  The 

problem  is  to  either 

identify  or  create  an 

infrastructure  to  capitalize 
on  these  resources.  There 
are  several  approaches  which 
are  under  investigation: 


b.  Can  Evaluate  before 
utilization  (try  before  we 
buy). 

c.  Possibly  better 
training. 

DISADVANTAGES: 

a.  Danger  of  either 
overtraining  or  under¬ 
training. 

b.  Training  may  be 

inconsistent. 

c.  Training  may  occur 
in  small  increments  over 
extended  periods. 

2.  Have  schools  custom 

tailor  instruction  to  meet 

specific  training  needs. 

ADVANTAGES: 

a.  Eliminates  Over  or 

Under  training. 

b.  Can  ensure 

consistent  training. 

c.  Most  time  sensitive. 

DISADVANTAGES: 

a.  More  costly. 

b.  Reduced  training 

sites. 
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3.  Use  Off-Shelf  Commercial 
Programs  to  Train. 

ADVANTAGES: 

a.  Most  cost-effective. 

b.  Unlimited  Training 

sites. 

c.  Most  time  sensitive. 

d.  Most  consistent. 

DISADVANTAGES: 

a.  Unable  to 

custom-tailor  to  specific 
needs. 

b.  Not  total  MOS,  only 
Modules. 


administrative  and 

legislative  constraints.  A 
feasibility  study  on  the  use 
of  various  VOTEC  models  is 
currently  being  performed. 
Phase  1  of  this  effort  will 
determine  which  models  can 
be  implemented  and  the 
methodology  for  doing  so. 
Phase  2  will  conduct  pilot 
programs  using  the  approved 
models.  The  four  models 
include: 

1.  High  School  Model 

2.  Preaccession  Model 

3.  Industry  Model 

4.  Post  Accession  Model 

See  Figure  3. 


4.  Distribute  VOTEC 

Training  from  the  Public 
Sector. 

ADVANTAGES: 

a.  Unlimited  Training 

sites. 


b.  Can  be  time 

sensitive. 

c.  Consistent  Training. 

DISADVANTAGES: 

a.  Costs  may  be 

prohibitive. 

b.  Distribution  System 
may  not  be  in  place. 

Each  of  the  alternatives 
could  be  adapted  to  various 
models  which  depict  when, 
where,  and  how  training  will 
be  delivered  within 


High  School  Model  One  of 
the  more  novel  concepts  is 
for  the  Army  to  mold  the 
vocational  training  programs 
in  the  high  schools  to  help 
meet  our  training  needs. 
This  concept  would  be  most 
feasible  for  light  green, 
low  tech  military 

occupational  specialties. 

Related  VOTEC  programs 
already  exist  in  many  of  the 
nations  high  schools.  In 
others,  a  military  science 
vocational  technical 

curriculum  could  be 

introduced  in  the  high 
schools.  This  would  be  an 
extension  of  the  Junior  ROTC 
program.  While  this 

programs'  focus  is  on 
building  citizenship.  Army 
Regulations  authorize  the 
development  of  such  a 
vocational  track. 
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Under  this  proposed 

scenario,  military  science 
would  be  offered  as  an 
alternative  to  traditional 
VOTEC  training  in  Junior  and 
Senior  years  of  high  school. 
It  is  envisioned  that 
a  junior  in  high  school 
would  be  presented  various 
Military  Vocational  Options 
from  the  local  recruiter.  A 
student  would  receive  skill 
level  one  training  along 
with  the  normal  mandatory 
high  school  subjects.  The 

Military  Science  VOTEC  could 
also  address  generic  hard 

skills  associated  with  the 
operation  and  maintenance  of 
military  systems.  Under 

provisions  of  the  Technology 
Transfer  Act,  the  Army  could 
provide  the  curriculum  and 
training  support  materials 
to  the  schools.  In  the 
future  this  would  include  a 
high  school  link  up  to  an 

Army  Distributed  Training 
Network. 

As  an  incentive,  the 

recruiter  could  offer  a 
"summer  job  option",  using 
existing  federal  programs, 
to  those  willing  to  sign  a 
letter  of  intent  to  enlist 
after  graduation.  (Could  be 
either  active  or  reserve 
commitment).  During  the 

summer,  student  could  attend 
Basic  Training  and/or 

participate  in  a  on-the-job 
training  program  with  either 
an  active  or  reserve  unit. 
This  would  provide  an 

opportunity  to  hone  those 
skills  learned  during  their 
junior  year.  During  this 

period,  they  would  draw  E-1 
pay  to  begin  and  E-2  pay  at 
the  end  of  the  summer  or  the 


BCT  phase.  In  their  senior 
year,  they  would  resume  the 
program.  Upon  graduation, 
they  could  be  promoted  to 
E-3  and  PCS  directly  to  the 
unit  for  a  two-year  enlist¬ 
ment.  It  is  envisioned  that 
they  would  enter  the  service 
with  most  Skill  Level  Two 
tasks  trained.  Introduction 
of  such  a  program  has  both 
direct  and  intrinsic 

benefits: 

a.  For  the  Army: 

—  Provides  a 
competitive  edge  with 

industry  to  better  tap  a 
dwindling  recruitment  pool. 

--  Provides  for  PCS 
directly  to  units  after 
graduation. 

--  Reduces  the 
resident  training  base  and 
the  costs  associated  with 
training  skill  level  one 
tasks. 

Creates  an 
infrastructure  of  civilian 
who  are  pro-Army. 

b.  For  the 

individual: 

--  Instills  self- 
discipline  at  a  critical 
time  in  the  adolescents 
maturation  process. 

--  Has  the  potential 
of  reducing  the  high  school 
drop-out  rate. 

Provides  job 
opportunities  for  various 
minorities. 


366 


Bolster 

economically  deprived  areas 
and  reducing  racial  tension. 

See  Figure  4. 


Preaccession  Model  (Delayed 
Entry)  As  the  force 

structure  declines  it 

appears  that  the  delayed 
entry  program  will  become 
the  norm.  An  architecture 
could  be  developed  which 
would  capitalize  on  this 
"dead  time"  by  providing 
training.  This  training 

could  be  structured  or 
simply  made  available  at 
military  learning  centers  or 
recruitment  centers. 

VOTEC  TRAINING 


o  Unstructured 

Installation 

Learning  Center 

--  Take  Home  Instruction 
(i.e.  Off-Shelf  electronics 
courseware) 

o  Structured 

—  Attends  VOTEC 
--  Establish  Recruiter/ 
MEP  Learning  Centers 


See  Figure  5. 

Indust'^'v  Model  The  industry 
model  (train  now,  pay  latet) 
takes  advantage  of  existing 
training.  It  suggests  that 
the  Army  can  recruit  those 
already  possessing  some  or 
all  the  skills  associated 
with  skill  level  one 
training  in  certain  MOS. 
This  could  be  accomplished 


through  the  development  or 
purchasing  of  achievement 
tests  to  certify  competency 
in  selected  skills  prior  to 
recruitment.  Tuition 

reimbursement  and  advanced 
promotions  could  be  offered 
as  enticements  for  enlist¬ 
ment.  In  some  MOS  the  Army 
should  be  able  to  compete 
favorably  with  industry  to 
recruit  recent  VOTEC 

graduates.  Some  examples  of 
MOS  which  fall  into  this 
category  inc’’  ide  all  19  band 
MOS  and  the  military  oolice 
MOS  (95B10). 


■ee  Figure  6- 


Post  Accession  Model  After 
an  individual  comes  on 
active  duty  several  options 
exist  for  utilizing  VOTEC. 
A  soldier  could  be  seru  to  a 
VOTEC  center  for  training  in 
his  local  community  prior  to 
BCT.  After  completing  the 
VOTEC  training  he  would  PCS 
to  his  PCT  station  then  to 
his  unit.  A  variation  could 
occur  where  an  individual  is 
sent  to  BCT  than  PCS  to  his 
first  assignment  where  VOTEC 
training  could  be  used,  if 
available,  to  train  the 
MOS.  Both  options  have 

monetary  advantages  over  the 
current  system.  Still 

another  option  would  have 
the  VOTEC  portion  of 
training  occur  after  BCT  but 
at  the  same  locale.  All  the 
options  above  save 

substantia?  costs  in  PCS 
moves  over  our  current 
system. 


367 


Contract  Training  Many  of 
the  models  above  include 
some  aspects  of  contract 
training.  The  training  for 
various  technical  skills 
such  as  diesel  mechanics, 
heavy  equipment  operators, 
nursing,  language,  etc.  are 
"light  green"  and  could  be 
economically  contracted 

out.  Contract  training 

could  occur  at  service 
schools,  corporations,  or 
public  or  private 

institutions.  Some  concern 
has  been  raised  that  lET 
soldierization  would  be  lost 
if  such  a  system  were 

adopted.  This  could  be 

avoided  by  contracting  for 
the  technical  training  prior 
to  basic  combat  training 
paving  the  way  for  direct 
PCS  to  the  unit.  (This 

presupposes  that  most 
soldierization  occurs  in 
BCT.)  Soldierization  could 
also  be  preserved  by 
assigning  personnel  to  an  RC 
Unit  for  basic  soldiering 
training,  (i.e.,  care  and 
feeding  of  the  uniform  and 
for  command  and  control) . 

Upon  completion  of  technical 
training,  an  individual 
would  complete  basic 

training  and  either  report 
directly  to  a  unit  for 

"greening"  of  the  technical 
training  in  the  functional 
context  of  a  unit/tactical 
environment. 

Certain  benefits  can  be 
derived  from  adopting  such  a 
system.  In  our  current 

system,  personnel  have  three 
difficult  simultaneous 

learning  experiences.  They 
are  learning  an  MOS  and  how 


to  be  a  soldier.  At  the 
same  time,  they  are  learning 
a  foreign  language;  military 
and  technical  jargon.  The 
requirements  of  soldieriza¬ 
tion  are  often  physically, 
mentally,  and  emotionally 
taxing.  This  hampers  a 

soldier's  ability  to  learn. 
In  adopting  the  concept 
proposed,  soldiers  would 
concentrate  on  developing 
technical  proficiency  in  a 
non-threatening  environ¬ 
ment.  It  would  also  reduce 
our  operational  costs  at  the 
service  schools  and  TOY  and 
PCS  expenses.  In  many 

cases,  these  savings  would 
offset  the  cost  of  the 
contract  training.  Improved 
accountability  for  training 
would  also  result  as  this 
would  be  a  factor  for  the 
awarding  or  maintaining  of  a 
contract. 

As  a  final  consideration, 
the  services  provided  by  the 
light  green  CMFs  could  be 
readily  contracted  out  to 
the  civilian  sector  and 
certain  MOS  eliminated  from 
the  active  force  structure. 
As  a  wartime  buffer,  a 
provision  to  the  contract 
could  require  a  certain 
percentage  of  the  personnel 
providing  the  services  to  be 
members  of  the  Reserve 
Component.  Thus,  a 

sustainment  environment  is 
created  where  personnel  in 
the  reserves  practice  their 
wartime  skills  on  a  daily 
basis.  (A  variation  of  this 
theme  currently  exists: 
Civilians  hired  by  the 
reserve  in  many  cases  luust 
be  reservists.) 
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Summary  From  a  personnel 
management  perspective,  many 
of  our  current  conventions 
need  to  be  reassessed. 
Trainers,  recruiters,  and 
personnel  managers  must  work 
more  closely  in  developing 
"cradle-to-grave"  continuums 
which  provide  maximum  return 
for  training  dollars 

invested. 

Innovative  recruitment 

procedures,  focusing 

recruitment  on  "low-cost" 
MOS,  creating  MOS  merger 
patterns  which  transition 
personnel  to  higher  cost  MOS 
during  their  second 

enlistments  can  all  result 
in  training  savings. 

New  screening  instruments, 
which  measure  both  cognitive 
and  psychomotor  achievement/ 
abilities,  will  better  match 
the  individual  to  the 
occupations  and  reduce 
reenlistment  retraining 

costs.  In  an  era  of 

constrained  budgets, 

policies  and  procedures  to 
reduce  low  reenlistment 
rates  for  many  MOS  must  be 
developed.  This  will 

promote  a  maximum  return  on 
our  investment  in  front-end 
trai'.j'ng  costs. 

As  the  size  of  the  Army 
decreases,  the  need  for 

higher  guality,  more 

experienced  personnel 

increases.  In  this  regard, 
the  current  "up  or  out" 
policy  needs  to  be 

reexamined  as  does  the 

"Specialist  Rank."  There 

appear*^  to  be  a  need  for  the 
career  technician  especially 


in  the  high  tech  CMFs  where 
proficiency  is  gained 

through  years  of  experience. 

By  leveraging  recruitment, 
the  Army  should  recruit 
individuals  with  as  many  of 
the  entry  skills  and 
knowledges  as  possible  to 
reduce  the  front-end  costs 
of  initial  entry  training. 
Focal  point  of  these 
recruitment  efforts  should 
be  in  the  high  density 
low-tech,  inexpensive  to 
train  CMFs. 

Finally,  although  the  square 
was  originally  conceived  as 
a  vehicle  to  model  training 
strategies  it  has  proven 
useful  in  other  respects. 
During  the  analysis  of  the 
MOS  characteristics,  it 
became  apparent  that 

differences  in  the  leader¬ 
ship  types  and  styles 
associated  with  supervising 
personnel  in  the  various 
occupations  should  exist. 
The  psychological  profiles 
of  leaders  in  each  of  the 
squares  also  seem 

different.  Future  efforts 
may  yield  a  leadership  task 
inventory  corrolated  to  the 
requirements  of  each 

quadrant.  The  square  has 
also  been  used  to  focus  at 
least  at  a  macro-level  the 
types  of  technologies  which 
we  should  be  exploring  with 
each  group  of  CMFs.  For 
instance,  artificial 

intelligence  training 

technologies  maybe  overkill 
for  training  purposes  for 
some  of  the  CMF  which  have 
low  technical  requirements. 
Generally  speaking,  the  need 
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for  more  sophisticated 

training  aids,  devices, 
simulations,  and  simulators 
increases  in  a 

counterclockwise  direction 
as  one  moves  around  the 
various  quadrants.  Again, 
further  work  is  warranted  in 
the  arena  to  determine  the 
utility  of  using  the  square 
to  focus  technological 

applications. 

The  paper  reflects  the  views 
of  the  author  and  should  not 
be  construed  as  Official 
Department  of  the  Army 
Policy  or  Doctrine. 
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CMF-CAREER  MANAGEMENT  FIELD 
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FIGURE  2 
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FIGURE  3 
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FIGURE  4 


FIGURE  5 


WHAT  DOES  THE  RESEARCH  LITERATURE  TELL  US  ABOUT  ADOPTING 

INNOVATIVE  TECHNOLOGIES? 


P.  Kelly  Watson,  Ph.D. 

The  study  of  the  adoption  and  use  of  innovation  and  technology  is 
relatively  new,  most  of  the  research  being  30  to  40  years  old. 
However,  there  seems  to  be  no  single  "grand  theory"  that  explaihs 
a  significant  portion  of  the  variance  found  in  studies  of  the 
adoption  of  innovations.  The  research  studies  are  often  either 
too  general  and/or  descriptive  in  their  results  and  methods,  or 
too  specific  to  generalize  to  other  settings.  There  are,  however, 
some  findings  that  seem  to  have  relevance  for  many  developers  and 
users  of  new  technologies  and  innovations,  regardless  of  the 
setting.  The  purpose  of  this  paper  is  to:  1)  provide  a 
definition  and  description  of  the  process  of  innovation  and 
change;  2)  provide  the  results  of  research  studies  of  adoption 
and  use  of  innovations,  especially  those  in  training  and 
education,  including  specific  examples  from  military  settings; 
and  3)  provide  a  set  of  guidelines  generated  from  the  research 
findings  that  can  be  used  to  plan  for  adoption  and  use  of 
innovations  and  new  technologies. 

The  process  of  creating  innovations  and  getting  people  to  use 
them  is  the  process  of  changing  behavior,  often  on  a  mass  scale. 
Therefore,  the  research  results  covered  in  this  paper  will  be 
presented  as  they  relate  to  behavioral  change,  using  the  stimulus 
-organism  -  response  -  consequence  (S-O-R-x)  model  as  a  typology. 
Thus,  the  research  findings  and  examples  will  be  presented  as 
stimulus  attributes  (e.g.,  attributes  of  innovations, 
characteristics  of  change  agents) ;  as  organizational  and 
individual  attributes  (e.g.,  personality  characteristics; 
cultural,  social  and  organizational  factors);  and  as  behavior 
attributes  and  consequences  (e.g.,  implementation  and  use, 
rewards  and  punishments  for  use) .  Guidelines  for  ensuring 
successful  change  or  adoption  will  be  presented  as  planning 
questions,  formatted  within  the  same  typology  that  was  used  to 
present  the  research  findings.  These  questions,  when  combined 
with  context-specific  criteria,  could  provide  the  basis  for  a  set 
of  procedures  for  the  systematic  planning  and  design  of  the 
innovation  adoption  process. 

P.  KELLY  WATSON,  Ph.D.  An  educational  psychologist  with 
Continental  Systems  Technology  Corporation  in  Marietta,  GA.  He 
has  worked  on  several  training  systems  programs  there,  and  was 
Training  Coordinator  for  Advanced  Training  Technology  for 
Lockheed  Aeronautical  Systems  Co.  prior  to  his  current  position. 
He  obtained  his  doctorate  from  Florida  State  University  in  1981. 
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WHAT  DOES  THE  RESEARCH 
LITERATURE  TELL  US  ABOUT 
ADOPTING  INNOVATIVE 
TECHNOLOGIES? 

P.  Kelly  Watson,  Ph.D. 

Introduction.  The  field  of 
education  has  produced  the 
greatest  number  of  studies  on 
the  dissemination  and  use  of 
knowledge  and  innovation,  but 
perhaps  no  other  field  has 
benefitted  less  from  the 
process .  Although  the 
quantity  of  studies  is  great, 
the  quality  has  been  decidedly 
less  than  hoped  for.  In  part, 
the  reasons  for  this  lack  of 
contribution  have  been 
associated  with  the  unique 
properties  of  educational 
organizations  and  procedures, 
the  variety  and  types  of 
"change  agents",  and  the 
nature  of  the  user  groups. 
Training,  as  a  subset  and 
special  case  of  education, 
shares  some  of  the  same 
problems.  In  some  cases,  the 
problems  are  exacerbated  by 
the  fact  that  training  shares 
the  bureaucratic 
characteristics  of  both 
educational  and  non- 
educational  organizations. 

There  are  features  of  the 
research  on  innovation  and 
change  that  are  applicable  to 
education  and  training 
settings,  however.  Dill  and 
Friedman  (1979),  in  their 
review  of  innovation  and 
change  literature,  have  noted 
that  the  characteristics  that 
describe  change  in  education 
are  essentially  the  same  as 
those  that  describe  change  in 
other  disciplines.  What 
varies  is  the  methodological 
rigor,  the  terminology,  and 


the  theoretical  framework 
associated  with  the 
researchers.  Rogers  and 
Shoemaker's  definitive  work  on 
change  and  innovation  (1971) 
suggests  that  there  will 
ultimately  be  a  convergence  of 
research  orientation  in  this 
field  (the  "middle  range 
analysis")  that  will  occur  due 
to  the  diffusion  of 
information  across  and  among 
disciplines.  Because  the 
research  tends  to  be  either 
very  theoretical  and 
descriptive,  or  very  empirical 
and  lacking  in  conceptual  and 
theoretical  structure,  Rogers 
and  Shoemaker  have  proposed 
the  generation  of  research 
that  results  in  propositions 
midway  between  specificity  and 
generality.  These 
propositions  may  then  be  used 
to  bridge  the  gap  between 
theory  and  empirical  findings, 
and  may  provide  a  means  of 
integrating  the  diverse 
research  traditions  found 
across  various  disciplines. 

This  paper  represents  an 
attempt  at  this  middle  range, 
in  that  the  framework  and 
concepts  presented  are  derived 
from  various  theoretical 
positions,  while  the 
importance  and  contribution  of 
the  concepts  are  results  of 
empirical  studies  that  support 
the  inclusion  of  the  concepts. 
The  paper  is  centered  around 
the  concept  of  innovation  as 
it  represents  planned, 
purposive  change.  This  change 
is  often  in  the  form  of  new 
technologies,  such  as  computer 
technologies.  When  people  or 
organizations  fail  to  adopt 
innovations,  however,  they 
usually  do  so  for  specific 
reasons.  Therefore,  the  real 
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study  of  innovation  is  the 
study  of  resistance  to  and 
acceptance  of  change. 

Innovation,  as  planned  change, 
is  intended  to  bring  about 
improvements  in  individuals  or 
social/organizational  systems. 
The  concept  of  planned  change 
assumes  that,  in  addition  to  a 
recipient  of  such  efforts, 
there  are  those  individuals 
and  organizations  (known  as 
change  agents)  that  attempt  to 
instill  these  changes  in  such 
recipients  (known  as  clients 
or  users) ,  and  that  these 
agents  meet  with  varying 
degrees  of  success  or  failure 
due  to  several  factors  (to  be 
addressed  in  this  paper) .  It 
is  important  to  note,  however, 
that  for  an  innovation  to  be 
perceived  as  truly 
"innovative",  it  must  be 
perceived  as  new  and/or  unique 
by  members  of  the  organization 
or  group  who  comprise  the 
group's  adopting  unit.  In 
some  instances,  the  "newness" 
of  an  innovation  may  be 
nothing  more  than  a  new, 
previously  non-existent 
attitude  toward  the  idea  or 
innovation  that  is  expressed 
by  the  adopting  group,  but 
even  this  has  implications  for 
whether  the  group  may  accept 
the  change  or  reject  it. 

The  Process  of  Change.  There 
are  several  paradigms  used  to 
describe  the  change  process. 

In  simple  terms,  Rogers  and 
Shoemaker  (1971)  suggest  that 
the  change  process  is  one  of 
communication,  in  which  a 
"sender"  (S)  gives  a  message 
(M)  through  a  particular 
channel  or  medium  (C)  to  a 
receiver  (R) ,  resulting  in 
some  effect  or  consequence  (E) 


(the  S-M-C-R-E  model) .  This 
model  has  been  adapted  in  this 
paper  to  correspond  to  a 
traditional  stimulus-organism- 
response-consequence  learning 
model  (S-O-R-x) ;  thus,  sender 
characteristics,  message  or 
innovation  type,  and  media  are 
subsumed  under  stimulus 
characteristics  (S) ,  receiver 
characteristics  are  subsumed 
under  individual  and 
organizational/ social 
variables  (O) ,  and  effects  are 
covered  by  the  response  and 
consequences  of  the  response 
(R)  . 

If  the  communication  process 
is  intended  to  bring  about 
purposeful  change  such  as  the 
adoption  of  a  new  technology, 
it  may  be  described  as  a 
sequence  of  stages  in  which 
the  innovation  is  either 
subsequently  adopted  or 
rejected.  Rogers  and 
Shoemaker  describe  the  stages 
of  adoption  as  follows: 

1)  Knowledge  -  Awareness  of  an 
innovation  is  created  and  the 
adopter  acquires  some 
understanding  of  the 
functioning  of  the  innovation. 
Preceding  this  stage  are  those 
individual  and  organization 
constructs  that  affect,  in 
theory,  the  further  processing 
of  the  change/adoption  process 
(e.g.,  individual  personality 
traits,  social, 
organizational,  and  cultural 
characteristics,  needs  of 
adopters,  etc.). 

2)  Persuasion  -  Attitudes  are 
formed  about  the  innovation  as 
a  result  of  the  information 
acquired. 

3)  Decision  -  An  individual  or 
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organization  forms  a  decision 
concerning  adoption  or 
rejection;  the  type  of 
decision-making  process  (e.g., 
democratic  vs.  authoritarian) 
is  a  major  determinant  of 
acceptance. 

4)  Confirmation  -  Adopters 
seek  reinforcement  for  a 
decision  concerning  adoption 
which  may  result  in  continued 
use  or  discontinuance  of  the 
innovation. 

Hall,  Loucks,  Rutherford,  and 
Newlove  (1975)  developed  a 
model  that  describes  the 
extent  to  which  teachers  use 
innovative  materials.  The 
model,  called  the  "Levels  of 
Use"  model,  entails  eight 
levels  within  seven  distinct 
stages  that  describe  a 
continuum  of  use.  The  self- 
explanatory  stages  are:  non¬ 
use,  orientation,  preparation, 
mechanical,  routine  and 
refinement  (two  levels  within 
the  same  stage) ,  integration 
(with  ongoing  methods) ,  and 
renewal.  The  model  assumes 
that  some  innovation  has  been 
introduced  to  the  instructor's 
organization  or  school,  and 
that  resistance  to  or 
acceptance  of  the  innovation 
can  be  described  by 
demonstrated  behavior  of 
teachers  or  instructors  that 
use  the  method  or  product  to 
varying  degrees. 

Similar  to  the  Levels  of  Use 
model  is  the  model  of  "Stages 
of  Concern",  also  developed  by 
Hall  and  Loucks  (1978) .  As 
with  the  Levels  of  Use  model, 
innovation  usage  is  described 
and  classified  after  it  has 
already  been  adopted.  This 
model  was  developed  to  attempt 


to  identify  attitudes  and 
concerns  of  teachers  regarding 
the  use  of  innovations,  rather 
than  as  a  means  for 
classifying  their  extent  of 
use.  In  this  model,  the 
stages  are; 

1)  Awareness  -  No  knowledge 
about  specific 
characteristics . 

2)  Information  -  Seeking  and 
receiving  information  about 
the  innovation. 

3)  Personal  -  Concern  about 
how  the  use  of  the  innovation 
will  affect  the  potential 
user. 

4)  Management  -  Concern  about 
the  extent  of  time  spent 
using/supporting  the 
innovation. 

5)  Consequence  -  Concern  about 
the  impact  of  use  on  end-user 
(e.g.,  students). 

6)  Collaboration  -  Concern 
about  relating  what  is  known 
about  the  innovation  with 
peers. 

7)  Refocusing  -  Concern  about 
changes  and  enhancements  to 
the  innovation  and  its  use. 

Although  the  orientation  of 
the  two  Hall  and  Loucks' 
models  are  levels  of  use  and 
expressed  concerns  rather  than 
the  change  process  itself, 
there  are  obvious 
similarities.  In  addition  to 
the  rough  parallel  to  the 
events  described  in  Rogers  and 
Shoemaker's  model,  the  stages 
described  in  both  models  also 
describe  empirically-derived 
stages.  The  use  of  "stage" 
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approaches  is  common  in  the 
literature  on  innovation  and 
change,  and  is  an  attempt  to 
categorize  the  common  steps 
that  occur  when  individuals 
and  organizations  are  faced 
with  purposive  change.  Figure 
1  shows  the  models  described 
above  and  their  process  and 
stage  similarities.  Perhaps 
the  simplest  description  of 
the  processes  involved  in 
change  and  adoption  of 
innovation  is  provided  by 
Stoffer,  Blaiwes,  and  Brictson 
(1980) ;  their  "acceptance 
process"  model  shows  a 
"unfreezing  -  changing  - 
refreezing"  sequence  of 
behavior  that  must  occur 
during  the  process  of  change 
(adapted  from  Lewin,  1951) . 
Their  conceptualization  of  the 
change  process  will  be  covered 
later  in  this  paper  as  a  means 
of  summarizing  some  of  the 
issues  involved  in  the 
adoption  of  innovations. 

In  addition  to  the  study  of 
change  as  a  function  of 
individual  behavior,  there  has 
been  a  considerable  effort  to 
study  change  as  a  function  of 
organizations  or  social 
systems.  Katz  and  Kahn 
(1966),  for  example,  view  the 
change  process  as  much  more  a 
province  of  organizations  than 
of  individuals  alone.  Their 
conceptualizations  have  shown 
that  organizational  factors 
must  be  included  as  variables 
contributing  to  the  change  or 
adoption  process.  They  cite 
as  examples  of  their  point  the 
common  practice  of  taking 
managers  and  foremen  out  of 
organizations  in  order  to 
receive  training  in  some  new 
method  designed  to  increase 
productivity,  improve  worker 


relations,  or  both.  When 
these  methods  fail  to  bring 
about  the  desired  changes,  the 
authors  charge,  the  reasons 
are  often  due  to  an  attempt  to 
make  changes  only  at  the 
individual  level  without  a 
corresponding  attempt  to 
address  organizational  factors 
at  the  same  time. 

Whereas  factors  related  to 
individuals  and  their  adoption 
and  use  of  innovation  are 
relatively  specific  and 
empirically  derived, 
organizational  variables  are 
often  general,  harder  to 
validate,  and  thus  less 
subject  to  control  and 
prediction.  However,  there  is 
sufficient  evidence  to  suggest 
that  they  have  importance  to 
the  study  of  adoption  and  use. 
The  middle  range  analysis 
described  earlier  may  also 
provide  a  means  of 
incorporating  specific 
individual  variables  with  more 
general  organizational 
variables  as  a  means  of 
explaining  more  of  what  is 
observed  in  real-life 
settings,  if  for  no  other 
reason  than  because  reality  is 
often  too  complex  to  be 
explained  by  one  approach. 

The  guidelines  provided  at  the 
end  of  this  paper  should  help 
by  offering  planners  of  change 
a  list  of  variables  (as  well 
as  an  organizing  framework)  to 
use  in  designing  the 
dissemination  of  new 
technologies . 

The  implications  of  the 
previous  points  are  that  many 
variables  may  be  linked  to 
adoption  or  rejection  of 
innovations.  Those  variables 
that  have  the  most  empirical 


380 


support  and/or  relevance  will 
be  presented  in  this  paper. 

The  S-O-R-x  paradigm  will 
include  attributes  of 
innovations  and 
characteristics  of  change 
agents  (S) ;  social, 
organizational,  and  individual 
characteristics  (O) ;  and 
innovation  implementation  and 
consequences  of  use  (R-x) . 

The  final  section  of  the  paper 
will  suggest  methods  of 
increasing  the  probability 
that  a  particular  innovation 
will  be  successfully 
implemented,  and  will  include 
the  guidelines  for  planners. 

Stimulus  Attributes  of 
Innovations  Rogers  and 
Shoemaker  (1971)  have 
succinctly  pointed  out  that 
the  adoption  or  acceptance  of 
an  innovation  is  related  to 
the  characteristics  of  the 
innovation,  not  as  seen  by 
experts  but  as  perceived  by 
potential  adopters.  They  have 
analyzed  the  literature  on 
change  across  several 
disciplines  and  have  generated 
distinct  categories  of 
attributes .  The  attributes 
are  listed  below,  and 
discussed  in  subsequent 
sections. 

1)  Relative  Advantage  -  the 
degree  to  which  an  innovation 
is  perceived  as  being  better 
than  the  idea  it  supersedes; 
often  measured  by  variables 
such  as  cost-benefit  ratio, 
labor  requirements  or 
increases,  perceived  risk, 
etc. 

2)  Compatibility  -  the  degree 
to  which  an  innovation  is 
perceived  as  consistent  with 
the  existing  values,  past 


experiences,  and  needs  of  the 
receivers. 

3)  Complexity  -  the  perceived 
quality  of  difficulty  in 
understanding  and/or  using  the 
innovation. 

4)  Trialability  -  the  extent 
to  which  an  innovation  may  be 
tried  on  a  limited  or 
temporary  basis;  reversibility 
(Zaltman,  Florio  and  Sikorski, 
1977)  is  one  facet  of  this,  in 
that  an  innovation  is 
irreversible  if 
discontinuation  of  its  use  is 
difficult  or  impossible; 
divisibility  of  innovations  is 
another  facet  that  increases 
the  probability  of  adoption. 

5)  Observability  -  the  extent 
to  which  the  results  of  an 
innovation  are  observable, 
noticeable,  or  communicable  to 
potential  adopters  (thus 
supporting  the  finding  that 
material  innovations  are  more 
readily  adopted  than  are 
nonmaterial  ones  such  as  ideas 
or  procedures) . 

Other  attributes  suggested  as 
determinants  of  adoption  aj.e 
the  radicalness  of  the 
innovation  (i.e.,  how  novel, 
creative,  and  risky  the 
innovation  is  and  how  much  of 
an  impact  it  is  likely  to  have 
on  its  users) ,  and  the 
scientific  status  of  the 
innovation  (Zaltman,  Duncan, 
and  Holbek,  1973;  Havelock, 
1979) .  This  latter  factor  is 
especially  important  for 
educational  and  training 
innovations,  because  it 
involves  the  perceived  worth 
of  the  innovation  in  terms  of 
its  validity,  reliability, 
internal  consistency,  and 
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empirical  support.  Perhaps  one 
of  the  more  famous  (or 
infamous)  variables  is  the 
point  of  origin  of  the 
innovation:  where  does  it  come 
from?  The  "not  invented  here” 
syndrome  is  an  example  of 
resistance  often  demonstrated 
for  innovations  whose  points 
of  origin  are  outside 
organizations . 

In  his  extensive  review  of 
factors  affecting  utilization 
of  innovations,  Burkman  (1987) 
suggests  that,  of  the  factors 
described  above,  the  most 
important  one  for  training  and 
instruction  innovations  is 
relative  advantage.  He  lists 
the  issues  related  to 
perceived  relative  advantage 
from  two  different 
perspectives :  the  instructor ' s 
and  the  organization's 
(specifically,  the  decision¬ 
makers  ' )  . 

Instructors  tend  to  view 
innovations  and  their  relative 
advantage  according  to:  1)  the 
amount  of  work  associated  with 
the  use  of  the  innovation,  and 
2)  the  effect  of  use  of  the 
innovation  on  their 
relationships  with  learners. 
Therefore,  instructors  may 
tend  to  reject  or  modify 
innovations  under  the 
following  circumstances: 

o  If  using  the  innovation 
might  reduce  instructor- 
student  interaction,  or 
involve  self-instruction, 
since  instructors  tend  to 
prefer  personal  interaction 
with  students  and  group 
instruction. 

o  If  there  is  a  increased 
management  function  associated 


with  the  innovation,  since 
instructors  believe  that  they 
have  little  extra  time 
available  for  scheduling, 
record-keeping,  etc. 

o  If  there  are  materials 
associated  with  the 
instructional  innovation  that 
are  perceived  as  inappropriate 
for  the  student's  level  or 
that  lack  motivational 
features,  since  instructors 
want  to  motivate  their 
students  to  perform  better. 

o  If  the  materials  are 
perceived  to  be  of 
insufficient  quality  to 
achieve  the  stated  objectives 
(usually  an  intuitive 
decision) . 

Decision-makers  within 
educational  and  training 
organizations  use  somewhat 
different  attributes  to 
determine  relative  advantage, 
as  might  be  expected.  They 
may  reject  or  modify  an 
innovation  under  the  following 
circumstances : 

o  If  the  cost  of  the 
innovation  is  out  of  the  scope 
of  the  budget  (probably  the 
most  common  reason)  or  if  the 
use  of  the  innovation  requires 
additional  resources  such  as 
per sonne 1 ,  equ ipment ,  e tc . 

o  If  the  innovation  will  not 
enjoy  a  wide  degree  of 
acceptance  among  users  in  the 
organization,  or  among  the 
organization's  benefactors  and 
clients  (this  is  closely 
allied  with  the  issue  of 
compatibility,  previously 
mentioned) . 

o  If  the  innovation  is  not 
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perceived  to  be  of  sufficient 
quality  to  achieve  student 
learning  (i.e.,  to  be 
efficient  and  effective) . 

Johnson  (1988)  has  provided 
support  for  the  kinds  of 
issues  likely  to  be  factors  in 
adoption  and  use  described  in 
Burkinan's  review.  In  his 
review  of  the  problems 
associated  with  fielding  more 
intelligent  tutoring  systems 
(ITS's),  Johnson  cites  four 
categorical  reasons  (combined 
below  into  two  major 
categories)  why  few  ITS's 
survive  the  transition  from  R 
&  D  settings  to  practice: 

o  Resources  -  shortages  in 
qualified  personnel,  as  well 
as  necessary  hardware  and 
software,  are  often  combined 
with  insufficient  funding 
(most  ITS  development  is 
funded  through  R  &  D  funds, 
which  are  notoriously 
inadequate) ;  these  problems 
affect  both  the  development  as 
well  as  the  implementation  of 
the  ITS. 

o  Attitudes  -  both  developers 
and  sponsors  often  display 
attitudes  about  the  quality 
and  amount  of  the  ITS 
developed  that  suggest  less 
than  adequate  accountability 
for  their  products,  which  may 
in  turn  lead  to  lower  quality 
products . 

These  two  kinds  of  problems 
would  be  predicted  by 
Burkman's  review  as  the  kind 
of  problems  or  issues 
associated  with  rejection  of 
ITS's  or  other  technological 
innovations  by  user  groups 
such  as  instructors  and 
managers . 


Another  example  of  resistance 
can  be  seen  in  a  study  by 
Evans  (1968).  The  author 
presents  a  case  study  of  the 
resistance  to  instructional 
television  (ITV)  by  faculty 
members  at  a  metropolitan 
university.  The  results  of 
this  investigation  showed  that 
resistance  (as  measured  by 
responses  to  attitude 
questionnaires  and  surveys) 
was  increased  by  four  major 
perceptions  of  ITV: 

1)  Compatibility  -  Most 
faculty  members  expressing 
negative  attitudes  believed 
that  ITV  was  not  compatible 
with  "good  teaching"  practices 
which  they  believed  required 
direct  student-faculty 
interaction. 

2)  Complexity  -  Resistant 
faculty  members  believed  that 
ITV  was  too  complex  and 
required  training  and 
technical  expertise  they 
lacked. 

3)  Divisibility  -  Some 
faculty  members  stated  that 
they  would  only  be  interested 
in  ITV  to  the  extent  that  it 
could  be  broken  down  into 
palatable  bits  and  used  as  an 
adjunct  to  on-going  teaching 
activities. 

4)  Point  of  origin  - 
Acceptance  and  rejection  of 
ITV  was  often  found  to  depend 
on  whether  faculty  members 
perceived  the  idea  to  have 
originated  within  their  own 
departments  (as  part  of  their 
own  planning  efforts) ,  or 
imposed  on  them  by  the 
university  administration. 

Attributes  of  innovations  are 
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often  brought  to  the  attention 
of  potential  users  by  the 
deliberate  efforts  of  change 
agents.  Change  agents  nay  be 
defined  most  simply  as 
individuals  who  strive  for  the 
improvement  of  some  agency, 
institution,  or  group  that 
they  believe  would  likely 
benefit  from  the  adoption  of 
an  innovation.  Therefore, 
what  people  perceive  as  the 
attributes  of  new  products, 
ideas,  or  technologies  are 
often  directly  the  result  of 
active  participation  by  some 
change  agent,  either  within  or 
outside  of  the  organization  or 
group.  (Note:  Earlier 
mention  was  made  of  channels 
of  communication  and  their 
role  in  change.  Of  the  two 
major  channels  of 
communication  -  interpersonal 
and  mass  media  (newspapers, 
television,  radio)  -  change 
agents  are  more  apt  to  be 
successful  in  implementing 
change  when  using 
interpersonal  channels  of 
communication.  For  this 
reason,  the  focus  of  this 
section  will  be  on  the  use  of 
interpersonal  channels  by 
change  agents  rather  than  on 
mass  communication.) 

What  makes  people  who  act  as 
change  agents  successful? 
Watson  (1981)  has  reviewed  the 
characteristics  and  actions  of 
successful  change  agents, 
presented  below: 

1)  Change  is  almost  directly 
related  to  the  level  of  effort 
put  forth  by  the  change 
agents:  the  more  effort,  the 
greater  the  chance  of  success. 

2)  Change  is  more  likely  to 
occur  when  change  agents 


perform  according  to  the  role 
expectations  of  the  clients 
rather  than  the  change 
agencies . 

3)  Change  is  more  likely  when 
the  advocated  solution  or 
innovation  is  perceived  to 
address  a  real  problem  rather 
an  assumed  one. 

4)  Successful  change  agents 
are  empathetic,  credible,  and 
share  similar  attributes  and 
traits  with  their  clients. 

5)  Change  agents  meet  less 
resistance  and  are  more  often 
successful  if  they  first 
approach  opinion  leaders 
rather  than  non-leaders  within 
the  organizations  in  which 
change  is  introduced. 

6)  Change  agents  are  more 
successful,  both  immediately 
and  in  the  future,  if  they 
increase  the  client's  ability 
to  evaluate  innovations 
(evaluation  ability  is 
positively  correlated  with 
proper  usage  of  the 
innovation) . 

7)  Resistance  to  change  and 
innovation  is  lessened  when 
change  agents  involve  their 
clients  in  the  development  or 
creation  of  the  innovation  or 
change,  if  they  anticipate  and 
address  sources  of  resistance 
before  they  arise,  and  if  they 
point  out  the  long  term 
benefits  of  the  proposed 
innovation  to  the  clients. 

8)  Resistance  to  change  is 
lessened  where  change  agents 
can  work  as  a  team  rather  than 
as  individuals. 

In  summary,  what  change  agents 
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do  is  to  Introduce  and  work 
for  the  adoption  and  use  of  a 
new  product,  idea,  or 
procedure.  They  are  most 
effective  in  accomplishing 
change  when  they  are  members 
of  the  organization  in  which 
change  is  introduced,  when 
they  reflect  personal 
characteristics  that  admired 
and/or  shared  by  users,  if 
they  involve  users  to  some 
degree  in  developing  and  using 
the  innovation,  and  if  they 
are  proactive  in  identifying 
and  resolving  potential 
problems  with  adoption  and  use 
of  the  innovation.  All  of 
this  assumes  that  the 
innovation  that  is  being 
promoted  is  an  acceptable 
solution  to  a  real  problem  or 
need  in  the  organization. 

Before  moving  to  the  next 
category  of  variables,  it 
should  be  mentioned  that  the 
discussion  of  the  innovation 
and  change  process  assumes 
what  Rogers  (1976)  calls  a 
"pro- innovation  bias".  That 
is,  the  research  literature 
and  the  approach  taken  to  the 
issue  of  change  assumes  that 
change  is  good,  that  those  who 
do  not  accept  the  innovation 
"resist"  or  "reject"  some 
presumably  good  development. 
This  is  obviously  not  always 
the  case,  but  it  is 
characteristic  of  Western 
cultures  and  societies  that 
are,  to  a  large  extent, 
products  of  technology. 

Change  and  new  technology  are 
everyday  aspects  of  life  here, 
but  not  in  other  countries  or 
even  all  socioeconomic  groups 
in  a  given  Western  country  or 
society.  What  often  happens, 
therefore,  in  studies  of 
innovation  is  that  there  is  a 


presumption  that  failure  to 
adopt  and  use  new  products  or 
procedures  is  due  to  an 
inherent  attitude  of 
resistance  to  all  new  things, 
or  due  to  some  other 
individual  flaw  in  the 
personality  of  the  potential 
user.  The  next  section  looks 
at  both  individual  variables 
and  organizational/social 
variables  that  may  be 
associated  with  resistance  to 
or  adoption  of  change  and 
innovation. 

Organizational  and  Individual 
Variables  In  the  previous 
section,  cultural  differences 
both  between  and  within  groups 
were  suggested  as  reasons  why 
innovations  are  rejected  or 
not  used.  Such  culture  "gaps" 
may  also  be  found  in 
education,  the  military,  or 
industry,  where  organizations 
and  groups  are  described  by 
their  own  organizational 
cultures.  The  construct  of 
"organizational  culture"  is 
one  that  can  be  used  to 
characterize  most 
organizational  and  social 
variables  affecting  acceptance 
and  rejection  of  innovation, 
including  social  norm's  and 
expectations,  grou’"  cohesion, 
and  the  function  the 
organization  in  society.  One 
of  the  more  difficult  problems 
with  describing  organizational 
barriers  to  adoption  and  use, 
however,  id  the  fact  that  the 
variable':^  exist  as 
hypothetical  constructs,  i.e., 
there  is  usually  no  way  to 
directly  observe  and  validate 
their  existence  or  their 
importance .  However ,  these 
variables  are  at  least 
valuable  methods  for 
describing  what  seems  to  be 
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occurring  in  organizations 
faced  with  the  introduction  of 
innovation.  The  variabler 
that  have  the  most  relevance 
to  resistance  and  acceptance 
of  innovation  follow. 

Because  all  organizations 
create  and  maintain  their  own 
barriers  to  other 
organizational  influences, 
separate  cultures  are  often 
created,  reflecting  their 
unique  characteristics.  This 
culture  may  result  in  its 
members  viewing  other  cultures 
as  inferior  to  theirs,  and 
this  sense  of  superiority  may 
often  intentionally  or 
unintentionally  be 
communicated  to  others. 

Sarason  (1971)  has  discussed 
this  problem  in  the  context  of 
schools,  and  notes  that  it  is 
most  common  to  the  university 
researcher  -  school 
practitioner  relationship. 
Because  they  view  the  cultural 
milieu  of  public  schools  as 
inferior  to  their  own  highly 
academic,  prestigious  culture, 
researchers  often  look  down  on 
teachers  and  administrators 
and,  as  a  result,  experience 
resistance  to  university 
research  and  its  results. 
Moreover,  this  problem  is 
found  in  other  organizations 
in  which  user  groups, 
practitioners,  and  clients  are 
recipients  of  research  studies 
intended  to  bring  about 
changes  in  their 
organizations.  Thus,  culture 
gaps  that  exist  because  of  the 
intentional  segregation  of  one 
culture  from  others  can  result 
in  problems  when  there  is  an 
attempt  to  communicate  or 
provide  assistance  to  others. 
Even  in  those  academic 
settings  in  which  such 


distinctions  are  minimal, 
there  may  still  be  a  culture 
gap  if  researchers  or 
innovators  fail  to  include 
their  clients  in  the 
developmental  process  of 
creating  change  (Zaltman, 
Florio,  and  Sikorski,  1977) . 
When  such  exclusion  occurs, 
clients  may  find  it  easy  to 
disregard  or  reject  the 
innovation  since  there  is  no 
ownership  of  the  plan  (cf. 
Drucker,  1973) . 

Havelock's  analysis  of  the 
change  process  is  useful  for 
describing  how  organizations, 
as  systems,  may  block  or 
inhibit  innovation  use 
(Havelock,  1979) .  When  viewed 
as  a  system,  the  organization 
functions  by  input  of 
materials  and  information. 
Thus,  the  first  point  of 
resistance  may  be  the  blocking 
of  information  into  the 
organization.  Some  of  the 
variables  that  may  explain 
information  blocking  are 
listed  below: 

o  Desire  for  stability  -  Most 
organizations  require 
stability  in  order  to  function 
properly;  innovative  practices 
are,  almost  by  definition, 
destabilizing  since  they  call 
for  the  implementation  of  new 
ideas  and  methods.  Stability 
is  a  factor  that  seems  to  act 
as  an  informational 
"gatekeeper"  in  that  it  is  a 
value  acting  as  a  barrier  to 
change.  When  the  desire  for 
stability  yields  lethargy  and 
insulation,  however,  it  may 
result  in  the  organization's 
becoming  entrenched  in  the 
technological  Dark  Ages. 
Perelman  (1990) ,  for  example, 
offers  an  anecdote  concerning 
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the  reaction  of  school 
administrators  to  an 
interactive  video  system  that 
could  increase  productivity  in 
adult  learning  settings  by  30% 
and  decrease  time  reguired  to 
complete  courses:  the 
administrators  rejected  the 
system  because  "the  district 
pays  us  for  attendance,  not 
achievement. ... if  anything  my 
ADA  (average  daily  attendance) 
might  go  down  and  my  budget 
could  get  cut". 

o  Coding  scheme  barriers  - 

All  organizations  create  their 
unique  language  and  jargon. 
When  organizations  want  to 
block  innovational  input,  they 
may  use  coding  scheme  barriers 
(i.e.,  communication  barriers 
such  as  recognition  of 
specific  terminology  only)  as 
a  means  of  rejecting  potential 
innovations. 

o  Financial  conditions  - 
Havelock  notes  that 
organizations  that  are  not 
financially  secure  are  often 
resistant  to  innovation 
because  they  can't  afford  the 
expense  or  resource  commitment 
to  make  changes.  On  the  other 
hand,  Havelock  reports  that 
financially  secure 
organizations  are  often  those 
that  are  most  open  and 
receptive  to  innovation. 

o  Training  and  staff 
development  -  Organizations 
that  provide  formal  training 
and  development  for  their 
employees  often  teach  the 
status  quo,  and  instruct 
"appropriate”  attitudes  for 
resisting  change. 

o  Organizational 
vulnerability  -  Sieber  (1967) 


describes  educational 
organizations  (and,  by 
extens ion ,  tr a ining 
organizations)  as  vulnerable 
organizations.  Vulnerability 
is  defined  as  the  "probability 
of  being  subjected  to 
pressures  that  are 
incompatible  with  one's  goals 
without  the  capacity  to 
resist".  Because  training  and 
education  organizations  serve 
clients  within  the  larger 
organization  or  community, 
they  must  be  responsive  to  the 
constantly  changing  demands  of 
their  clients,  even  if  the 
wishes  of  the  client  are  at 
odds  with  the  goals  and 
procedures  of  the  serving 
organization.  Such 
adjustments  to  the  demands  of 
the  client  create  strain  and 
tension  within  the 
organization,  and  there  is  a 
resultant  lag  between  what  the 
organization  does  and  what  is 
expected  of  it.  Such  lags 
create  even  more  pressure  for 
change.  The  result  of  such 
conflict  is  twofold;  1)  a 
general  resistance  to 
innovation  and  change  overall, 
and  2)  an  acceptance  of  those 
innovations  or  changes  desired 
by  client  organizations, 
almost  regardless  of  merit. 

Resistance  may  also  occur  as  a 
result  of  the  internal 
functioning  of  the 
organization  -  the  system 
"throughput”,  in  systems 
theory  terminology.  This 
refers  to  the  flow  of 
information  and  the  major 
processes  that  are  routinely 
ongoing.  The  decision  to 
innovate  or  adopt  some  change 
in  procedures  is  the  focal 
point  around  which  the 
variables  related  to 
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resistance  are  spread.  The 
most  important  variables  are 
listed  and  described  below: 

o  Type  of  decision-to- 
Innovate  -  In  formal 
organizations,  decisions  made 
concerning  the  adoption  of 
innovations  are  typically  of 
two  kinds:  authority 
innovation-decisions .  and 
collective  innovation- 
decisions.  As  the  name 
implies,  authority  decisions 
are  made  by  those  in  positions 
of  power  and  imposed  on  the 
organization,  while  collective 
decisions  are  made  by 
consensus  of  group  members. 
Rogers  and  Shoemaker  (1971) 
have  stated  that  the  attitude 
of  the  adopting  unit  toward 
the  innovation  is  highly 
related  to  the  type  of 
decision  process  used  to 
implement  the  change: 
authority  innovation-decisions 
tend  to  generate  resistance 
among  users,  as  well  as  what 
Rogers  and  Shoemaker  call 
innovation  dissonance.  This 
concept,  like  cognitive 
dissonance  (cf.  Festinger, 
1957) ,  suggests  that  there  is 
a  discrepancy  between  the 
behavior  required  by  users  of 
an  innovation  and  the  user's 
attitudes  toward  the 
innovation.  Dissonance  theory 
predicts  that  either  the 
behavior  or  the  attitude  would 
change  as  a  result  of  such 
dissonance;  reducing  the 
dissonance,  then,  often  leads 
to  discontinuation  of  use  of 
the  innovation,  or  at  best  a 
reduction  of  use  overall. 

o  Status  differences  among 
groups  -  In  those 
organizations  in  which  there 
are  differences  in  status 


among  the  groups  comprising 
the  organization,  there  is 
likely  to  be  a  status 
hierarchy.  In  such 
organizations,  high  status  or 
authoritative  members  are  less 
likely  to  accept  information 
if  it  comes  from  those  lower 
in  the  organizational 
hierarchy,  despite  the  fact 
that  those  lower  in  the 
hierarchy  may  be  more  in  touch 
with  critical  issues  of 
performance  than  high  status 
members.  In  some  cases,  whole 
groups  enjoy  higher  status 
than  other  groups,  and  this 
relationship  can  impede  the 
process  of  adoption  or  use  of 
innovations.  A  good  example 
of  group  status  differences  as 
barriers  to  change  can  be  seen 
in  large  defense  contractor 
corporations.  In  most,  if  not 
all,  of  these  organizations, 
the  most  prestigious  and 
highest  status  group  is  the 
engineering  division.  Thus, 
engineering  organizations 
often  dictate  inappropriate  or 
incomplete  solutions  to 
problems,  as,  for  example,  in 
cases  in  which  the  project  is 
a  training  development 
project.  Since  most  training 
organizations  historically  are 
low  in  organizational  status, 
training  projects  are  designed 
and  developed  according  to 
engineering  principles  rather 
than  training  needs  and 
requirements.  This  results  in 
disregard  for  training 
information  or  innovative 
approaches  if  they  are  not 
part  of,  or  consistent  with, 
the  engineering  domain. 

o  Existing  job  roles  -  Most 
organizations  reward  personnel 
only  for  what  they  are  doing 
now.  Existing  tasks  and  jobs 
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typically  define  the  roles  of 
workers  but  do  not  provide 
rewards  or  support  for 
innovation  or  new  ways  of 
functioning.  This  essentially 
encourages  workers  not  to  rock 
the  boat.  Again,  Perelman 
(1990)  offers  an  example  of 
this  kind  of  barrier.  He 
interviewed  teachers  in  one  of 
the  country's  most  affluent 
school  districts  to  determine 
why  there  had  been  so  little 
teacher  interest  in  a  large 
program  involving  computer- 
aided  instruction.  One 
teacher  responded  "  Why  should 
I  do  anything  different  next 
year  from  what  I  did  last 
year?  Who  cares?"  Part  of 
this  attitude  was  a  result  of 
the  school  bureaucracy's 
failure  to  support  or  reward 
innovation  in  the  classroom. 

Psychological  studies  have 
provided  many  possible 
variables  and  concepts  that 
could  be  used  to  explain  why 
people  change  or  fail  to 
change  their  behavior.  In 
social  psychology,  for 
example,  consistency  and 
balance  (McGuire,  1972;  Heider 
1958)  have  been  identified  as 
motivating  factors  in  humans 
to  explain  why  people  tend  to 
behave  in  ways  that  minimize 
drastic  changes  to  their 
lives.  Maslow  (1954) 
attempted  to  explain  the 
desire  for  stability  and  need 
fulfillment  by  suggesting  a 
hierarchy  of  needs  that  humans 
strive  to  meet.  The 
individual's  needs,  combined 
with  a  general  desire  to 
maintain  a  physical  and 
psychological  steady  state, 
may  lead  to  the  individual ' s 
acceptance  of  those  objects  or 
activities  in  the  environment 


that  meet  the  most  basic  or 
immediate  needs.  Thus,  the 
more  relevant  innovations  are 
to  specific  individual  needs, 
the  more  likely  they  are  to  be 
accepted  (Havelock  and  Benne, 
1967) .  Such  explanations 
support  the  well-known  concept 
of  homeostasis,  which 
suggests,  in  part,  that  humans 
seek  balance  and  minimal 
disruption  of  their  lives. 

Consistency  of  behavior, 
however,  does  not  suggest  that 
individuals  who  adopt  an 
innovation  will  continue  to 
adopt  other  innovations. 

Rogers  and  Shoemaker  (1971) 
have  stated  that  individuals 
tend  to  shift  from  adopter  to 
non-adopter  categories  with 
some  regularity,  making  it 
difficult  at  best  to  predict 
with  any  degree  of  certainty 
whether  an  individual  who  has 
accepted  an  innovation  will 
adopt  other  innovations  in  the 
future.  They  suggest  that 
consistency  of  innovativeness 
is  more  likely  to  be  seen  in 
individuals  adopting  consumer 
or  material  innovations  rather 
than  less  tangible  changes 
such  as  political  ideology. 

Some  of  the  specific 
psychological  variables  that 
are  useful  for  explaining 
acceptance  of  or  resistance  to 
change  are  listed  below.  This 
is  by  no  means  a  complete 
list,  but  is  useful  for 
identifying  some  of  the  more 
important  personality  factors 
associated  with  individuals 
and  their  general  approach  to 
change  and  innovation. 

o  Sense  of  competence  -  This 
factor  may  be  defined  as  the 
individual's  perception  of 
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what  he  or  she  can  and  cannot 
do,  based  upon  the  person's 
history  of  behavior  and 
accomplishment .  The 
importance  of  this  variable 
for  this  context  is  that  it 
may  be  a  predictor  of  the 
future  behavior  of  individuals 
with  respect  to  adoption  of 
innovation.  Those  having 
faith  in  their  abilities  to 
perform  tend  to  seek  out  more 
and  different  things  to 
accomplish  in  their 
environments,  including  the 
trial  of  innovations  and  new 
technologies.  Objects  and 
practices  that  are  perceived 
to  be  not  within  one's  area  of 
competence  are  more  likely  to 
be  rejected,  especially  if  the 
perceived  area  of  competence 
is  small  (Havelock,  1979) . 

o  Persuasibility  -  This 
general  factor  is  an  indicator 
of  an  individual's  tendency  to 
be  affected  by  deliberate 
influences.  Persuasibility  is 
affected  by  factors  such  as 
imagery  and  empathy,  "other- 
directedness"  orientations, 
social  extroversion,  and  a 
lack  of  overt  hostility. 
Persons  exhibiting  this  factor 
to  high  degrees  may  be  more 
prone  to  adopt  or  reject  an 
innovation  (Janis,  1963) . 

o  Authoritarianism  -  The  most 
global  of  indicators  is  the 
authoritarian  personality 
(Adorno  et  al.,  1950). 
Originally,  the  studies  that 
were  performed  on 
authoritarianism  were  directed 
at  specific  ethnic  groups  that 
were  believed  to  be  culturally 
oriented  toward  rigidity  and 
control.  Later  studies  showed 
that  all  cultures  can  and  do 
foster  the  development  of 


individuals  exhibiting  traits 
such  as  prejudice,  intolerance 
of  ambiguity,  rigidity, 
political  and  economic 
conservatism,  overconcern  with 
status  and  success,  and  a 
strong  sense  of  submission  to 
authority.  Such  individuals, 
according  to  the  researchers, 
are  "natural"  resisters  of 
change,  preferring  to  move 
slowly  and  cautiously  when 
confronted  with  new  ideas. 
Later  research  by  Rokeach 
(1960)  attempted  to  show  that 
authoritarianism  is  not 
extreme  political  beliefs,  and 
thus  should  rightly  be 
regarded  as  dogmatism  instead. 
The  implication  of  these 
studies  is  that,  to  the  extent 
that  these  personality  traits 
exist  and  are  demonstrated  by 
individuals,  more  resistance 
to  change  would  be  expected. 

Although  factors  and  traits 
like  those  described  above  may 
be  important  for  understanding 
some  of  the  dynamics  of 
humans,  they  shed  little  light 
on  how  we  may  intentionally 
change  such  conditions  to 
favorably  impact  adoption  of 
innovations.  The  burden  of 
promoting  change  will  continue 
to  be  on  creating  good,  useful 
innovations  and  practices, 
emphasizing  the  best 
utilization  attributes  in  the 
design  and  development  of  the 
innovation,  and  following  up 
on  the  use  after  adoption. 

This  last  category  of  adoption 
variables  -  what  happens  after 
the  decision  to  adopt  or 
accept  the  innovation  -  will 
be  the  topic  of  the  next 
section. 

Behavior  Attributes  and 
Consequences  of  Innovations 
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In  many,  if  not  most 
organizations,  there  may  be  a 
difference  between  the 
adopting  group  and  the  user 
group.  In  educational  and 
training  organizations,  for 
example,  administrators  and 
managers  may  decide  to  adopt 
an  innovation  or  technological 
practice  that  is  to  be  used  by 
teachers  or  instructors.  The 
organizational  environment  is 
therefore  ripe  with 
opportunities  for  resistance, 
since  the  decision  is  often 
made  without  the  input  of 
users.  The  two  major 
categories  of 
adoption/rejection  of 
innovations  during  this  stage 
are  1)  implementation  (use)  of 
the  innovation  and  2) 
consequences  of  the  use. 

Immediately  after  the  adoption 
decision  is  made,  provisions 
are  typically  made  for  the 
implementation  of  the 
innovation.  Fullan  and 
Pomfret  (1977)  have  made  an 
extensive  review  of  literature 
pertaining  to  the 
implementation  of  educational 
innovations,  especially  those 
that  are  curriculum 
innovations.  Although  user 
training  was  found  to  be 
linked  to  greater  rates  of 
use,  intensive  in-service 
training  was  found  to  be  the 
most  effective  means  of 
ensuring  continued  use  and 
acceptance.  Contrast  this 
approach  with  pre-service  or 
workshop  types  of  training,  in 
which  instructors  receive 
cursory  instruction  or 
familiarization  with  the 
materials  or  devices.  In- 
service  training  was  found  to 
be  more  effective  due  to  1) 
the  provision  of  detailed 


demonstrations ,  2 )  the 
provision  of  reinforcing 
experiences  in  hands-on 
situations  with  the  materials, 
and  3)  an  opportunity  for 
instructors  to  be 
“resocialized"  in  the  new  ways 
of  the  innovation's  use.  The 
addition  of  elements  such  as 
these  (i.e.,  beyond  training 
materials  alone)  is 
responsible  for  the  kind  of 
increased  learning  and 
performance  exhibited  by  the 
teachers  and  instructors 
studied  by  Fullan  and  Pomfret 
(see  Matthews  and  Fawcett, 

1977  and  Fawcett  and  Fletcher, 
1977  for  additional  examples 
of  how  intensive  training 
makes  innovation  use  more 
resistant  to  extinction) . 

The  implementation  issue  is 
really  a  two  sided  one.  If  an 
innovation  has  been  field- 
tested  and  validated  for  use 
among  its  intended  population, 
it  should,  in  theory,  be  used 
exactly  the  way  it  is  designed 
and  in  the  manner  in  which 
users  are  trained.  However, 
as  noted  earlier,  one  of  the 
attributes  of  "good" 
innovations  is  that  they  be 
adaptable  to  the  needs  of  the 
user.  This  means  that  users 
will  change  the  use  regimen  to 
suit  their  own  specific 
purposes.  Therefore,  the 
innovation  comes  to  be  used  in 
a  manner  different  from  its 
intended  use.  At  the  risk  of 
seeming  evasive,  it  must  be 
noted  that  the  answer  lies 
with  both  positions.  Good 
innovations  must  be  designed 
and  developed  with  user  input 
and  must  be  user-friendly; 
they  must  also  be  designed  to 
be  adaptable  to  the  growing 
needs  of  user  groups  as 
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necessary.  This  approach  can 
be  seen  in  the  formal  Levels 
of  Use  identified  by  Hall  et 
al.  (1975)  and  described 
earlier,  in  which  users  may 
ultimately  refine  and  change 
the  innovation  according  to 
their  needs. 

The  best  illustration  of  the 
two  sided  nature  of  innovation 
use  can  be  seen  with  the 
military's  use  of  the 
Instructional  Systems 
Development  (ISD)  model. 

Since  the  formal  development 
and  implementation  of  this 
approach  among  the  various 
branches  of  the  services  in 
the  mid  1970s,  the  model  has 
been  used  and  subsequently 
both  hailed  as  the  saving 
grace  of  systematic  training 
and  decried  as  a  terribly 
expensive,  time-consuming 
method  that  yields  results 
indistinguishable  from  older 
instructional  methods. 

Branson,  one  of  tne  authors  of 
the  Interservice  Instructional 
Systems  Development  model,  has 
ncced  that  the  system  can  be 
used  as  written  with  great 
success;  however,  the  model 
can  also  be  adapted  to  the 
particular  constraints  of 
users  without  fear  that  such 
adaptations  will  invalidate 
anything  created  using  the 
model.  The  ISD  model  was 
developed  to  provide 
courseware  developers  with 
sets  of  guidelines  for 
curriculum  development,  not  a 
Bible  that  must  be  followed  in 
an  orthodox  manner.  In  fact, 
since  the  original 
Interservices  ISD  model  was 
developed,  there  have  been 
several  changes,  additions, 
and  collateral  documents 
created  that  provide 


additional  guidance  (e.g.,  the 
Training  System  for 
Maintenance  -  TRANSFORM  - 
developed  by  the  Air  Force's 
3306th  Test  and  Evaluation 
Squadron) .  The  most  useful 
innovations  are  those  that 
work  when  used  as  designed, 
but  also  allow  for  adaptations 
(R.  K.  Branson,  personal 
communication,  December  1, 
1988)  . 

One  of  the  major  failings  of 
research  in  the  area  of 
innovation  adoption  and  use  is 
the  limitation  of  studies  only 
to  the  process  of  adoption 
rather  than  continuing  the 
investigation  to  include  what 
happens  after  the  innovation 
has  been  adopted.  Such 
studies  have  made  the  rate  of 
adoption  the  critical 
dependent  variable,  lather 
than  a  more  inclusive  set  of 
variables  related  to  the 
consequences  of  adoption  and 
use.  The  consequences  of  use 
of  new  materials  and  practices 
are  perhaps  more  difficult  to 
measure  and  may  not  be  of 
interest  to  some  researchers; 
nonetheless,  the  impact  may  be 
significant  especially  to 
individuals.  The  problem  for 
individuals  is  that  there  are 
rarely  any  rewards  or  them  to 
use  or  develop  innovations. 

The  best  description  of  what 
is  at  stake  is  provided  by 
Beales  (1968,  and  cited  in 
Kaufman,  1970) : 

"Technicians  and  planners 
often  seem  both  surprised  and 
distressed  when  resistances 
are  encountered  to  innovations 
of  obvious  merit  -  obvious,  at 
least,  in  the  eyes  of  the 
innovators.  Yet  in  fact 
resistance  to  change  is 
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normally  to  be  expected  at 
some  point  from  some  members 
of  tbe  receiving  society. 

Even  minor  technological 
innovations  not  only  may 
introduce  new  means  to  achieve 
established  ends  but  may 
create  new  goals.  In  addition 
to  requiring  the  establishment 
of  new  work  habits, 
technological  change  involves 
some  restructuring  of  social 
relationships.  A  relatively 
slight  change  may  mean  the 
obsolescence  of  one  valued 
skill  which  gave  status  or 
economic  security.  More 
extended  changes  may  destroy 
whole  security  systems  for 
part  of  a  population,  alter 
relative  statuses  and  economic 
positions,  and  redistribute 
power  and  leadership.  All 
individuals  and  groups  who 
perceive  change  to  be  to  their 
disadvantage  in  any  way  will 
be  resistant  to  them  unless 
compensating  advantages  are 
presented  or  evident.  Even  to 
admit  the  superiority  of  the 
knowledge  of  another  may 
involve  intolerable  loss  of 
prestige  or  self-esteem  in 
some  societies.  The 
technological  innovation  which 
sooner  or  later  arouses  no 
resistance  must  be  extremely 
trivial.**  (Emphasis  added,  p. 
170)  . 

Example  Studies  Two  studies 
are  provided  that  demonstrate 
the  relevance  of  the  variables 
described  herein.  The  first 
study  is  from  the  Florida 
educational  system,  and  shows 
the  importance  of  some  of  the 
adoption  and  use  factors  as 
perceived  by  user  groups.  The 
second  example  is  from  the 
military  training 
establishment  and  shows  the 


results  of  a  review  of 
variables  associated  with  the 
success  and  failure  of  self- 
paced  learning  courses  in  the 
Air  Force. 

Richardson  and  Papagiannis 
(Note  1)  reported  findings  of 
a  study  of  nine  facilitating 
conditions  of  innovation  use 
as  perceived  by  teachers  and 
administrators  in  23  Florida 
schools.  Table  1  shows  the 
results  of  ratings  of 
importance  of  the  conditions. 
It  should  be  noted  that  the 
two  most  important  conditions 
as  perceived  by  the  sample  of 
educators  were  organizational 
support  and  individual 
motivational  factors.  The 
rest  of  the  list  and  their 
associated  ratings  indicate 
that  proper  training  and  user 
input,  along  with  active 
change  agents,  are  necessary 
ingredients  in  the  innovation 
adoption  and  use  process,  (see 
Table  1) 

The  second  study  is  provided 
by  McCombs,  Back,  and  West 
(1984)  and  provides  an 
indication  of  the  importance 
of  utilization  variables  in 
the  design,  development  and 
use  of  self-paced 
instructional  materials.  The 
authors  reviewed  the  use  of  12 
self-paced  courses  at  four  Air 
Force  technical  training  bases 
to  identify  the  factors 
associated  with  successful  use 
or  failure  and  discontinuance. 
Table  2  presents  the  list  of 
factors  resulting  from  the 
study.  Motivation  of  the 
instructors ,  instructor 
training,  organizational 
support,  participatory 
development  and  management, 
and  flexibility  of 
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implementation  were  all  noted 
as  important  variables  in  the 
study.  Importantly,  the  kinds 
of  innovation  attributes  cited 
by  Burkman  (1987)  also 
received  support:  importance 
of  student- instructor 
interactions,  costs  of  and 
need  for  additional  equipment, 
and  quality  of  student 
materials  were  cited  as 
factors  determining  success  or 
failure. 

The  two  studies  cited  above 
indicate  that  the  kinds  of 
variables  identified  in 
studies  of  innovation  adoption 
and  use  (and  described  in  this 
paper)  are  found  in  specific 
educational  and  training 
settings  in  both  the  civilian 
and  military  sectors.  The 
majority  of  innovation  that 
occurs  in  civilian  settings, 
however,  is  curriculum-based, 
and  (with  the  exception  of 
computer-based  instruction) 
may  not  reflect  high  levels  of 
technology  such  as  those  found 
in  the  equipment  used  in 
military  settings.  The 
McCombs,  Back  and  West  study 
also  focused  on  instructional 
materials  as  the  source  of 
innovation.  Do  the  same 
issues  apply  to  settings  in 
which  innovation  is 
predominately  technological 
and  oriented  toward  computers 
and  electronics? 

Stoffer,  Blaiwes,  and  Brictson 
(1981)  suggest  that  the  answer 
to  this  question  is  yes.  In  a 
review  of  the  problems 
associated  with  user 
acceptance  of  research  and 
development  studies  and 
training  devices  in  the  Navy, 
the  authors  list  a  number  of 
issues  that  are  common  to  the 


variables  described  above. 

They  are  presented  below: 

o  Deficiencies  in  user 
motivational  conditions  - 

these  include  lack  of  rewards 
or  incentives  for  users, 
increased  workload,  and  poor 
relations  with  R  &  D  groups. 

o  Deficiencies  in  user  role 
assignments  -  this  relates  to 
the  differences  in  culture  and 
methods  associated  with  user 
and  R  &  D  groups  concerning 
issues  such  as  cost  of  the 
study,  need  for  the  study, 
failure  to  maintain  proper 
liaison  before  and  during  the 
study ,  etc . 

o  Deficiencies  in  official 
organizational  policy  and 
structure  -  this  includes  the 
implementation  of  "non¬ 
interference"  policies  that 
can  make  R  &  D  more  difficult, 
instructor  rotation/turnover , 
lack  of  instructor  training, 
and  lack  of  R  &  D  support. 

o  Inadequate  defense  R  &  D 
contracting  methods  -  there  is 
no  formal  recognition  of  user 
acceptance  as  a  problem  to  be 
addressed. 

o  Failure  to  include  users  in 
the  acquisition  process  - 
there  is  a  lack  of 
participative  management,  thus 
reducing  the  user's  sense  of 
ownership  in  the  process  of 
design,  development,  and  use. 

o  Other-than-rational  user 
responses  to  R  f  D  - 

expressions  of  resistance  to  R 
&  D  studies  due  to  perceived 
lack  of  applicability  to  the 
user's  setting,  complexity  of 
setting,  lack  of  credibility 
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of  researchers,  and  inadequacy 
of  training  devices. 

o  Deficiencies  in  the 
training  device  design  - 

inadequate  design  of  student 
station  (usually  associated 
with  lack  of  physical 
fidelity) ,  or  lack  of 
appropriate  instructional 
features. 

The  review  of  factors 
associated  with  user  rejection 
of  R  &  D  and  training  devices 
suggests  that  there  is 
considerable  overlap  between 
the  types  of  variables  listed 
as  causes  of  user  rejection 
(or  acceptance)  in  both  high 
technology  and  low  technology 
settings.  In  their 
conceptualization  of  the  user 
acceptance  process,  Stoffer, 
Blaiwes,  and  Brictson  have 
developed  a  model  that 
reflects  the  critical  issues, 
stages  of  change,  and  kinds  of 
change  to  be  achieved.  This 
model  is  presented  in  Figure 
2.  Their  conceptualization  is 
useful  for  showing  the  inter¬ 
relationships  of  the  degree  of 
change,  its  desired 
permanence,  and  the  kinds  or 
levels  of  change  desired.  The 
categories  of  the  kinds  of 
change  desired  are  derived 
from  Hersey  and  Blanchard 
(1977) ,  and  provide  a  roughly 
parallel  approach  to  the 
stages  of  the  various  models 
presented  in  Figure  l.  In  the 
Hersey  and  Blanchard  approach, 
change  is  perceived  as  more 
difficult  to  achieve  but 
becomes  more  permanent  as  it 
is  incorporated  as  a  part  of 
the  behavior  of  individuals 
and  groups.  There  is  also  an 
attempt  to  group  the  kinds  of 
change  with  the  stages  of 


change  and  the  critical  issues 
associated  with  completing  a 
study  or  accomplishing  the 
desired  change  (note  use  of 
vertical  dotted  lines  to 
indicate  commonalities) . 


Summary  and  Conclusions  This 
review  of  innovation  adoption 
and  use  has  provided  general 
variables  related  to  the 
change  process  and  specific 
variables  that  are  related  to 
education  and  training 
settings.  These  variables 
were  presented  using  a 
stimulus-organism-response- 
consequence  (S-O-R-x)  paradigm 
as  the  organizing  model  for 
grouping  and  presenting  the 
results  of  the  studies.  There 
is  consensus  that  the  results 
of  studies  on  innovation  and 
change  across  disciplines  are 
generalizable  to  educational 
and  training  settings. 

The  question  that  must  be 
answered,  however,  is  why 
innovation  adoption  and  use 
remains  a  problem  when  so  much 
is  known  about  the  causes  of 
user  rejection  and 
discontinuance  of  use.  The 
answers  are  probably  varied, 
ranging  from  lack  of  awareness 
of  the  issues  on  the  part  of 
developers  and  researchers,  to 
lack  of  resources  for  ensuring 
successful  adoption  and  use, 
to  outright  apathy.  For  those 
organizations  willing  to 
assist  the  innovation  use 
process,  the  results  of 
studies  such  as  this  one 
should  be  used  to  generate  a 
formal,  validated  method  of 
planning  for  change.  The 
first  step  in  such  a  process 
is  the  identification  of  the 
possible  variables  involved  in 
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the  adoption  and  use  process, 
which  is  accomplished  through 
papers  and  studies  such  as 
this  one.  These  results  can 
be  provided  to  planners  in  the 
form  of  "issues  to  be 
addressed"  until  such  formal 
processes  for  planning  are 
developed  and  tested.  Figure 
3  shows  the  kinds  of  questions 
that  should  be  addressed  when 
planning  for  the 
implementation  of  some 
innovation  or  technology, 
having  been  derived  from  the 
variables  identified  by  this 
review. 

In  addition  to  the  use  of 
planning  issues  shown  in 
Figure  3,  there  are  additional 
ways  in  which  we  can  help  to 
ensure  minimal  resistance  to 
innovations: 

1)  Develop  an  objective 
methodology  for  categorizing 
and  evaluating  innovations  - 
Both  Downs  and  Mohr  (1976)  and 
Burkman  (Note  2)  have 
suggested  that  the  worth  of 
new  technologies  and 
instructional  innovations  must 
be  determined  by  a  more 
objective  means  of 
categorizing  innovation 
characteristics  and  by  more 
objective  and  thorough 
evaluations  of  the 
effectiveness  of  the  process 
or  product  that  comprises  the 
innovation. 

2)  Document  the  need  for  the 
Innovation  and  ensure  that  it 
is  a  viable  solution  to  a 
documented  problem  -  Many  of 
the  instances  of  user 
rejection  of  innovations 
concerned  the  quality  of  the 
innovation,  and  its  ability  to 
be  used  effectively  in  the 


intended  setting  with  the 
targeted  group.  The  first  law 
of  applied  R  &  D  or 
instructional  development 
should  be  that  the  product 
meet  an  identified  need. 

3)  Provide  incentives  and 
support  as  part  of  the 
organizational  commitment  to 
innovation  to  ensure  that 
users  are  not  punished  for 
valuing  change  and 
innovation  -  In  schools  and 
school  systems  this  effort  may 
only  be  possible  through 
massive  restructuring  of  the 
system,  which  makes  this 
method  unlikely. 

4)  Provide  training  to  all 
users  of  the  innovation 
throughout  the  life  cycle  of 
the  innovation  -  This  will 
ensure  that  the  product  or 
method  is  implemented 
correctly  and  will  provide  the 
intended  results. 
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Figure  2:  Stoffer,  Blalwee.  and  Bricteon'e  model  of  the 
acceptance  ofoceae  (  19811. 
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USING  TRAFFIC  MODELS 
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MITRE 

Houston,  Texas 
Abstract 

The  proposed  Space  Station  Training  Facility  (SSTF)  at  the  NASA  Johnson 
Space  Center  will  contain  several  simulators,  Instructor/Operator  Sta¬ 
tions  (lOSs),  and  other  equipment.  Different  architectures  have  been  sug¬ 
gested  for  the  SSTF  and  are  to  be  evaluated.  A  single  Local  Area  Network 
(LAN)  has  been  proposed  for  inter-connecting  the  simulation  hosts  and  all 
the  lOSs.  Models  were  developed  to  simulate  the  data  traffic  anticipated 
on  the  network  due  to  both  the  cyclic  and  the  stochastic  interaction 
between  the  hosts  and  the  lOSs.  The  performance  of  different  LANs, 
including  Starlan®,  Ethernet®,  IEEE  802.5  token  ring  and  Fiber  Distributed 
Data  Interface  (FDD!)  was  assessed  under  a  variety  of  conditions  using  the 
models.  Besides  studying  normal  traffic,  capability  for  handling  burst 
traffic  was  evaluated.  The  study  showed  that  Ethernet,  token  ring  and 
FDDI  would  meet  the  requirements  but  they  differed  in  their  potential  for 
accommodating  growth  in  the  number  of  lOSs. 
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TRAINING  FACILITY  NETWORK 
ASSESSMENT  USING  TRAFFIC  MODELS 

Daniel  T.  Wick  and  Ankur  R.  Hajare 


INTRQPUCTIQM 

As  a  part  cf  the  Space  Station  Freedom  Pro¬ 
gram,  a  new  training  facility,  called  the 
Space  Station  Training  Facility  (SSTF),  is  to 
be  built  at  the  NASA  Johnson  Space  Center 
(JSC)  at  Houston,  Texas.  The  SSTF  is  in  the 
planning  stages  and  alternate  architectures 
are  being  evaluated.  Commonality  with 
existing  and  proposed  NASA  systems,  and  the 
use  of  commercial-off-the-shelf  products 
are  driving  forces  in  selecting  among  alter¬ 
nate  design  choices.  The  methods  being  used 
to  study  the  various  alternatives  include 
discrete  event  simulation  and  analytic  mod¬ 
elling.  One  of  the  studies  performed  was  an 
investigation  of  the  feasibility  of  a  commer¬ 
cial-off-the-shelf  Local  Area  Network 
(LAN)  for  all  the  Instructor/Oper-ator  Sta¬ 
tions  (lOSs)  witnin  the  SSTF 

There  are  at  least  four  valid  reasons  for  net¬ 
working  all  the  lOSs  instead  of  the  direct 
point-to-point  connections  that  have  been 
used  in  the  past  in  training  facilities  such  as 
the  Shuttle  Mission  Training  Facility.  First 
of  all,  networking  the  lOSs  permits  inter¬ 
changeability,  i.e.,  any  lOS  can  be  used  with 
any  host  computer.  In  the  Shuttle  Mission 
Training  Facility  this  was  achieved  by  means 
of  a  switch  for  the  two  Shuttle  Mission  Sim¬ 
ulator  (SMS)  bases  and  the  two  host  comput- 
ers[1].  In  the  SSTF,  where  at  least  fh  e 
simulators  and  13  lOSs  are  presently  under 
consideration,  a  switch  becomes  an  expen¬ 
sive  and  cumbersome  solution.  A  second 
reason  for  utilizing  a  network  approach  is 
that  it  supports  growth.  New  lOSs  and  hosts 
can  be  added  easily  to  a  network,  whereas 
such  additions  are  much  more  complex  with 
a  switching  arrangement.  This  is  important 
for  the  SSTF  because  the  number  of  trainers 
will  probably  change  during  the  projected 
long  life  span  of  Space  Station  Freedom. 

A  third  significant  advantage  in  utilizing  a 
network  solution  is  that  a  training  session 


with  multiple  lOSs  can  be  configured  easily 
on  a  network.  The  concept  of  operations 
document  for  the  SSTFI2]  calls  for  combined 
training  sessions  which  may  have  multiple 
lOSs.  With  a  net.vork,  any  subset  of  the 
available  lOSs  can  be  assigned  easily  to  any 
combination  of  simulators.  Finally,  a  net¬ 
work  solution  supports  message  broadcasting 
to  all  lOSs.  Though  numerous  direct  point- 
to-point  connections  could  be  used,  the  sim¬ 
plest  approach  to  the  broadcasting  require¬ 
ment  is  an  lOS  network. 

MODELLING  TOOLS 

Two  modelling  tools  were  used  for  this  study. 
Both  of  them  build  discrete  event  simulation 
models.  The  first  tool  was  the  Performance 
Analysts  Workbench  System®  (PAWS®), 
marketed  by  Scientific  and  Engineering  Soft¬ 
ware  of  Austin,  Texas.  Version  3.0  of  PAWS 
was  used  on  a  Digital  Equipment  Corporation 
VAX  11/750. 

The  second  tool  used  for  the  study  was  Net¬ 
work  11.5®  which  is  marketed  by  CACI 
Products,  Inc.  of  La  Jolla,  California.  Ver¬ 
sion  4.0  of  Network  11.5  was  used  on  a  IBM 
PS/2  Model  80.  Network  11.5  contains 
built-in  models  tor  transfer  devices  that  use 
collision,  token  ring  and  other  protocols.  A 
specific  LAN  is,  therefore,  modelled  by  an 
appropriate  selection  of  parameters. 

IPS  NETWORK  MODELS 

Models  were  developed  for  Ethernet,  Starlan, 
token  ring,  and  Fiber  Distributed  Data  Inter¬ 
face  (FDDI)  LANs.  All  of  them  are  described 
by  Stallings[3].  For  each  protocol  several 
models  were  built  in  order  to  assess  the 
performance  under  various  conditions.  Ini¬ 
tially,  a  Localtalk®  LAN  was  also  modelled. 
However,  it  was  soon  discarded  because 
Localtalk  has  a  data  rate  well  below  the 
expected  load. 

The  statistics  collected  for  each  model 
included  LAN  utilration,  queue  lengths,  and 
the  minimum,  maximum  and  the  average  of 
the  message  transfer  times. 
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Basic  Configuration 

Since  the  SSTF  is  still  in  the  planning  stages, 
the  system  architecture  has  not  been  final¬ 
ized.  Various  candidate  architectures  have 
been  proposed  and  one  of  them  was  selected 
for  modelling.  It  consists  of  five  host  com¬ 
puters  and  13  lOSs,  as  shown  in  Figure  1. 

The  traffic  on  the  network  was  estimated  on 
the  basis  of  the  preliminary  versions  of  the 
conceptual  design  and  the  concept  of  opera¬ 
tions  of  the  SSTF[2].  In  the  model,  lOSs  send 
two  types  of  messages  to  a  host  computer: 
(1)  commands,  and  (2)  requests  for  a 
downtoad,  e.g.  a  new  page.  These  messages 
were  assumed  to  be  80  bytes  long  and  40 
bytes  long,  respectively,  on  the  average. 
Commands  were  generated  with  a  Poisson 
distribution  with  a  mean  of  1  minute. 
Requests  for  a  download  were  also  generated 
with  a  Poisson  distribution,  but  with  a  mean 
of  10  minutes. 

Each  lOS  receives  a  message  from  a  host  once 
a  second  to  update  information  in  its  display 
pages.  It  was  assumed  that  1 0  display  pages 
would  be  updated  and  each  one  would  contain 
100  double  precision  variables.  The  mes¬ 
sage  length  for  this  was  estimated  at  72  Kb, 
which  would  include  numeric  values  as  well 
as  the  variable  text  on  a  page. 

Commands  from  the  lOS  to  the  host  do  not 
cause  any  additional  data  traffic.  Instead, 
they  cause  changes  in  the  data  that  is  con¬ 
tained  in  the  periodic  updates.  When  an  lOS 
requests  a  download,  a  host  responds  by 
sending  144  Kb  of  data.  The  download  has  a 
lower  priority  than  the  periodic  update  but 
cannot  be  preempted  once  it  has  started. 

The  data  traffic  used  for  the  model  is 
intentionally  higher  than  would  be  expected 
in  a  typical  scenario.  The  numbers  were 
chosen  in  order  to  add  a  factor  of  safety  to  the 
results  obtained  from  the  model.  However, 
the  data  rate  derived  above  is  still  consider¬ 
ably  less  than  the  data  rate  between  the  lOS 
'’nd  the  host  computer  of  the  SMS.  Mea¬ 
surements  on  the  SMSH)  indicated  a  traffic 
rate  750  Kb/second  to  the  lOS.  The  dif¬ 
ference  between  the  SMS  data  rate  and  the 


lower  rate  anticipated  in  the  SSTF  can  be 
explained  on  the  basis  of  architectural 
differences  ensuing  from  technological 
advances  since  the  design  of  the  SMS  in  the 
mid-1970s.  The  lOS  of  the  SMS  contains 
alphanumeric  and  graphic  terminals  with 
little  internal  processing.  The  entire  dis¬ 
play  is  re-drawn  during  an  update.  The  lOSs 
in  the  SSTF,  on  the  other  hand,  will  be 
workstation  based.  Page  formats  will  either 
be  stored  at  the  workstation  or  will  be 
downloaded  to  it  from  the  host.  The  entire 
display  will  not  have  to  be  re-drawn  every 
second.  Only  the  information  needed  to  up¬ 
date  the  displays  will  be  sent  to  the  lOSs. 

At  the  beginning  of  a  training  session  there 
may  be  file  transfers  from  the  host  to  the 
lOSs  in  addition  to  many  requests  for  a  down- 
toad.  This  was  not  simulated  because  these 
would  be  accomplished  before  the  real-time 
simulation  began  and  would,  therefore,  not 
impact  real-time  operation. 

Ethernet  Model 

Ethernet  is  a  Carrier  Sense  Multiple 
Access/Collision  Detect  LAN  running  at  a 
speed  of  10  Mb/second.  Stations  wishing  to 
transmit  listen  for  a  signal  and  transmit 
only  when  they  sense  that  the  LAN  is  idle.  If 
more  than  one  station  starts  transmitting,  a 
collision  is  detected  and  transmission  is 
aborted.  The  stations  attempt  to  transmit 
again  after  a  random  interval,  using  a  binary 
exponential  back-off  algorithm. 

Ethernet  was  modelled  with  a  fixed  size  col¬ 
lision  window,  although  in  reality  it  is  a 
function  of  the  distance  between  the  nodes 
that  are  attempting  simultaneous  transmis¬ 
sion.  In  order  to  study  the  performance  un¬ 
der  different  conditions,  the  model  was  run 
with  various  parameter  settings.  The  frame 
size  on  an  Ethernet  can  vary  from  a  mini¬ 
mum  of  46  bytes  of  data  per  frame  to  a 
maximum  of  1500  bytes  of  data  per  frame. 
The  two  extremes  were  modelled.  The  colli¬ 
sion  window  on  an  Ethernet  depends  upon  the 
length  of  the  cable  and  upon  the  repeaters  in 
the  path.  The  maximum  permissible  col¬ 
lision  window  is  51.2  ps.  which  is  the  time 
for  the  smallest  frame,  i.e.  46  bytes  of  data 
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Figure  1 

Modelled  Configuration 


plus  18  bytes  of  overhead.  The  model  was 
run  with  a  collision  window  of  51.2  ps  and 
with  a  collision  window  of  2  ps,  which 
corresponds  to  650  feet  of  cable  and  no 
repeaters.  This  is  reasonable  for  an  lOS  LAN 
within  one  buiiding,  as  will  be  the  case  for 
SSTF. 

The  model  was  run  for  one  hour  of  simulated 
time  for  each  scenario.  Initially,  the  model 
had  update  messages  arriving  at  exactly  one 
second  intervals.  Thus,  all  data  for  the  once 
per  second  update  was  queued  at  one  instant. 
The  model  was  then  modified  to  depict  a  more 
realistic  scenario.  This  consists  of  the  up¬ 
date  messages  arriving  sometime  within  each 
second  with  a  uniform  probability  distribu¬ 
tion. 

When  the  messages  arrived  with  a  uniform 
statistical  distribution,  the  number  of  colli¬ 
sions  decreased  to  a  relatively  small  num¬ 
ber.  Since  collisions  contributed  insignifi¬ 
cantly  to  utilization,  there  was  little  change 
to  the  utilization  of  the  LAN.  The  difference 
between  uniform  arrivals  and  instantaneous 
periodic  arrivals  was  in  the  message  trans¬ 
fer  time.  With  a  uniform  arrival  rate,  the 
average  message  transfer  time  was  only  8 
ms,  just  slightly  higher  than  the  theoretical 
best.  However,  with  instantaneous  periodic 
arrivals  it  was  20  ms.  The  maximum  was 
about  the  same  for  both  cases  because  a  sta¬ 
tistically  uniform  arrival  rate  could  create  a 
situation  as  bad  as  simultaneous  arrivals. 

Starlan  Model 

Starlan  is  a  Carrier  Sense  Multiple 
Access/Collision  Detect  LAN  like  Ethernet, 
but  has  a  date  rate  of  1  Mb/second.  The 
specifications  for  frame  size  and  collision 
windows  for  Starlan  are  the  same  as  for 
Ethernet,  hence  Starlan  was  modelled  with 
the  same  frame  sizes  and  collision  windows 
as  Ethernet. 

Token  Ring  Model 

All  stations  on  a  token  ring  LAN  are  con¬ 
nected  in  a  ring.  When  the  LAN  is  idle,  a 
token  circulates  around  the  ring.  A  station 
that  wishes  to  transmit  must  wait  until  it 


gets  the  token.  It  then  holds  the  token  until 
it  has  finished  transmission. 

The  IEEE  802.5  standard[4]  specifies  two 
data  rates,  1  Mb/second  and  4  Mb/second. 
Therefore,  both  data  rates  were  modelled 
even  though  the  lower  rate  is  not  in 
widespread  use.  A  higher  rate  that  is  com¬ 
mercially  supported,  16  Mb/second,  was  not 
modelled  because  it  was  not  a  standard  at  the 
time  of  the  study.  The  IEEE  802.5  standard 
does  not  directly  specify  a  maximum  frame 
size.  However,  the  maximum  time  that  a 
node  can  hold  a  token  is  10  ms  by  default. 
For  a  1  Mb/second  token  ring,  the  10  ms 
time  period  translates  to  a  frame  size  of 
1125  bytes  of  data  and  the  model  was,  there¬ 
fore,  run  with  that  frame  size.  The  IEEE 
802.5  standard  does  not  specify  a  minimum 
size  for  a  frame.  However,  the  model  was 
run  with  a  frame  size  of  46  bytes  of  data  to 
afford  a  comparison  with  Ethernet. 

For  a  4  Mb/second  token  ring,  the  default 
limit  of  10  ms  on  token  holding  time  trans¬ 
lates  to  a  maximum  frame  size  of  4500 
bytes  of  data  in  a  frame.  The  model  was  mn 
with  that  maximum  frame  size  and  also  with 
a  frame  size  of  46  bytes  for  the  sake  of  com¬ 
paring  it  to  Ethernet. 

FDDI  Model 

The  FDDI  LAN  is  a  100  Mb/second  LAN  with 
a  token  ring  protocol  that  is  similar  to  IEEE 
802.5.  Data  encoding  in  FDDI  is  different 
from  IEEE  802.5,  but  this  does  not  affect  a 
performance  model.  Token  passing  is  differ¬ 
ent  and  this  was  taken  into  account  in  the 
model.  Anomalies  are  handled  differently  in 
FDDI  and  IEEE  802.5,  but  these  special  situ¬ 
ations  were  not  modelled.  The  maximum 
frame  size  for  FDDI  is  4500  bytes  and  this 
frame  size  was  modelled.  Also,  a  frame  size 
of  46  bytes  was  modelled  for  a  comparison 
with  Ethernet. 

RESULTS 

The  results  of  the  model  runs  are  presented 
here  for  the  basic  configuration,  for  burst 
traffic  and  for  a  growth  scenario. 
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Basic  Configuration 

Under  the  best  conditions  (small  collision 
window,  large  frames),  the  Ethernet  LAN 
was  only  9.5%  busy.  Collisions  increased 
the  utilizations  by  less  than  0.01%.  There 
were  about  12  collisions  per  second  during 
the  simulation  run  but  each  one  used  the  LAN 
for  just  a  few  ps.  When  the  frame  size  was 
decreased  to  the  minimum  allowed  by  Ether¬ 
net,  the  frame  overhead  increased  the  LAN 
utilization  to  13.4%.  Once  again,  collisions 
affected  this  figure  by  less  than  0.01%. 

In  order  to  assess  performance  under  the 
worst  conditions,  the  size  of  the  coliision 
window  was  increased  to  the  maximum 
ailowable  for  Ethernet.  This  roughly  tripled 
the  number  of  collisions  and  increased  the 
utilization  from  13.4%  to  13.5%.  The  pri¬ 
mary  effect  of  the  higher  collision  rate  was 
not  on  utilization  but  on  the  message  trans¬ 
fer  time.  The  average  message  transfer  time 
increased  from  27  ms  to  31  ms.  However, 
the  maximum  message  transfer  time  during 
a  one  hour  run  increased  from  47  ms  to  83 
ms. 

The  model  demonstrated  that  Starlan  could 
not  handle  the  traffic  under  any  conditions. 
As  the  simulation  progressed,  message 
queues  kept  on  increasing,  demonstrating  the 
inability  of  the  LAN  to  handle  the  offered 
load.  The  same  condition  was  encountered 
with  the  1  Mb/second  token  ring  model, 
demonstrating  that  it,  too,  was  not  capable  of 
handling  the  traffic. 

The  4  Mb/second  token  ring  model  demon¬ 
strated  that  this  LAN  was  capable  of  handling 
the  workload.  This  LAN  was  34.7%  busy 
with  short  frames,  and  the  maximum  mes¬ 
sage  transfer  time  was  105  ms.  With  the 
maximum  size  frames,  the  LAN  was  23.9% 
busy,  and  the  maximum  message  transfer 
time  was  73  ms. 

The  FDDI  model  showed  that  this  LAN  was 
only  1.7%  busy  with  short  frames.  With  the 
maximum  size  frames,  the  LAN  was  only 
1 .0%  busy.  The  maximum  message  transfer 
time  was  oniy  about  1 0  ms. 


Burst  Traffic 

The  ability  of  the  different  types  of  LANs  to 
handle  burst  traffic  was  studied  by  simulat¬ 
ing  an  instantaneous  burst  of  100  messages 
of  72  Kb  each.  The  time  to  recover  from  the 
burst  was  compared  for  the  different  typ.^s 
of  LANs.  As  might  be  expected,  FDDI  was  the 
fastest  to  recover  from  the  burst,  taking 
only  0.1  seconds  to  do  so.  Ethernet  required 
1  second  to  recover  from  the  burst  and  a  4 
Mb/second  token  ring  took  3  seconds. 

Growth  Capacity 

The  capacity  for  growth  for  each  LAN  was 
studied  by  adding  lOSs  until  the  saturation 
point  was  reached,  indicated  by  message 
queues  increasing  in  length  as  the  simulation 
progressed.  Thus,  the  maximum  number  of 
lOSs  that  the  LAN  could  support  was  derived. 
Since  this  was  done  for  various  combinations 
of  parameters,  a  significant  number  of  model 
runs  was  required  to  determine  the  breaking 
point  for  each  case. 

As  the  network  load  Increased,  Ethernet 
showed  the  maximum  transaction  time 
increasing  rapidly,  even  though  the  average 
did  not  increase  much.  This  was  because  a 
few  transactions  suffered  many  collisions. 
Under  the  best  conditions,  Ethernet  could 
support  115  lOSs.  However,  when  the  col¬ 
lision  window  was  increased  from  2  ps  to 
51.2ps,  the  maximum  number  of  lOSs 
dropped  to  35.  Clearly,  under  a  heavy  load, 
the  size  of  the  collision  window  has  a  signifi¬ 
cant  effect.  Changing  the  frame  size  from  the 
1500  byte  maximum  to  the  46  byte  mini¬ 
mum  did  not  have  that  much  of  an  effect.  The 
maximum  number  of  lOSs  dropped  to  25 
when  the  frame  size  was  reduced  to  the  mini¬ 
mum. 

For  Starlan,  the  limits  to  acceptable  perfor¬ 
mance  were  studied  by  lowering  the  number 
of  tOSs  until  the  LAN  could  handle  the  load. 
Under  the  best  conditions  it  could  handle  12 
lOSs.  When  the  collisions  window  was 
increased  to  the  maximum,  it  could  only  han¬ 
dle  10  lOSs.  Under  the  worst  conditions 
Starlan  could  handle  only  6  lOSs. 
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To  determine  the  limit  for  a  1  Mb/second 
token  ring,  the  number  of  iOSs  was  reduced, 
as  with  Starlan.  With  long  frames  (I.e. 
1125  bytes  of  data),  a  1  Mb/second  token 
ring  couid  handle  12  IOSs.  However,  with  a 
frame  size  of  46  bytes  of  data,  it  could  han¬ 
dle  only  9  IOSs. 

The  growth  capacity  of  a  4  Mb/second  token 
ring  was  investigated  by  increasing  the 
number  of  IOSs  in  the  model,  as  with  Ether¬ 
net.  With  long  frames,  a  4  Mb/second  token 
ring  could  support  49  IOSs,  whereas  with 
short  frames  the  maximum  number  was  32. 

In  the  case  of  FDDI  number  of  IOSs  was 
increased  to  200  and  there  were  no  perfor¬ 
mance  problems.  Since  this  is  far  more  than 
the  number  of  IOSs  that  the  SSTF  will  ever 


have,  there  appeared  to  be  no  practical  rea¬ 
son  for  expending  further  effort  to  deter¬ 
mine  the  limit  at  which  an  FDDI  LAN  would 
be  saturated. 

The  growth  capacity  for  the  four  LANs  stud¬ 
ied  is  listed  in  Table  1. 

CONCLUSIONS 

The  models  demonstrated  that  Starlan  and  the 
1  Mb/second  token  ring  would  not  handle  the 
traffic  anticipated  on  the  lOS  LAN  in  the 
SSTF.  However,  Ethernet,  the  4  Mb/second 
token  ring  and  FDDI  were  capable  of  handling 
the  traffic.  All  three  offered  growth  poten¬ 
tial  well  beyond  the  requirements  antici¬ 
pated  today. 


LAN 

Number  of  IOSs 

minimum 

maximum 

Ethernet 

25 

115 

Starlan 

6 

1  0 

1  Mb  Token  Ring 

9 

12 

4  Mb  Token  Ring 

32 

49 

FDDI 

>  200 

>  200 

Table  1 :  lOS  LAN  Capacity 
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A  Training  Facility  for  Space  Station 
Astronauts 

Ankur  R.  Hajare,  James  R.  Schmidt  and 

Daniel  T.  Wick 

IhfTRODUCTION 

The  Space  Station  Training  Facility  (SSTF) 
will  be  located  at  the  Lyndon  B.  Johnson 
Space  Center  (JSC)  in  Houston,  Texas.  The 
SSTF  will  support  training  of  Space  Station 
astronauts,  Space  Station  Control  Center 
(SSCC)  ground  support  personnel,  and  Sta¬ 
tion  customers  throughout  the  life  of  the 
Space  Station  Freedom  Program  (SSFP). 

The  primary  objectives  of  the  SSTF  are  to 
provide  astronauts  and  ground  support  per¬ 
sonnel  with  the  generic  training  necessary  to 
allow  operation  of  the  Station  systems  and  to 
provide  Station  configuration-specific 
training.  SSTF  training  is  conducted  in  both 
normal  and  contingency  operations.  The 
generic  and  configuration-specific  training 
requirements  of  the  SSTF  are  summarized  in 
the  SSTF  requirements  document  (1  ]. 

Fundamental  to  the  training  philosophy  for 
the  SSFP  is  a  commitment  to  attain  com¬ 
monality  across  the  training  media,  the 
curriculum,  and  the  training  facilities.  In 
all  Instances,  this  commonality  will  have  to 
be  maintained  through  intensive  coordination 
between  the  centers  developing  the  Station, 
supporting  NASA  centers,  the  international 
partners,  the  scientific  community,  and  the 
academic  community. 

The  SSTF  is  conceptually  divided  into  the 
Author  Environment  and  Student  Environ¬ 
ment  [2],  both  of  which  may  be  supported  by 
capabilities  that  are  provided  by  the  SSFP  or 
by  NASA  institutional  resources. 

AUTHOR  ENVIRONMENT 

The  role  of  the  Author  Environment  is  to 
support  the  management  and  operations  of 
the  SSTF  and  to  support  the  development  of 
the  hardware  and  software  components  of 
training  loads  used  in  the  Student  Envi¬ 
ronment.  The  Author  Environment  contains 


the  systems  necessary  to  manage  the  facility; 
to  develop,  integrate,  test  and  verify  train¬ 
ing  loads;  and  to  develop,  reconfigure, 
maintain,  and  operate  the  hardware  and 
software  used  in  both  environments  of  the 
SSTF.  Primary  goals  of  the  Author  Envi¬ 
ronment  are  to  ensure  the  quality  of  the 
training  loads  used  in  the  Student  Environ¬ 
ment  and  to  verify  the  quality  and  fidelity  of 
the  systems  used  in  both  the  Author  Envi¬ 
ronment  and  the  Student  Environment. 

The  Author  Environment  systems  will 
address  all  phases  of  the  training  load 
development  life-cycle  from  design  through 
final  acceptance  testing.  The  systems  will 
provide  support  for  activities  such  as  relia¬ 
bility  testing  and  the  demonstration  of 
human  factors  design  principles.  Reliability 
and  availability  requirements  will  be  veri¬ 
fied  by  analysis  and  special  tests  such  as 
verification  of  redundant  components, 
examination  of  operations  trend  data,  and  the 
use  of  built-in  test  equipment  where  prac¬ 
tical. 

The  Author  Environment  systems  will  be 
used  to  perform  audits  of  software  quality 
attributes,  such  as  correctness  and  com¬ 
pleteness,  and  to  collect  the  data  required  to 
measure  hardware,  software,  system,  and 
subsystem  reliability  and  availability 
attributes.  Quality  assurance  and  common¬ 
ality  audits  will  be  conducted  throughout  the 
development  process.  These  audits  will  col¬ 
lect  and  organize  the  evaluation  data  at  var¬ 
ious  periods  during  the  requirements  defi¬ 
nition  and  product  design  phases.  Analysis  of 
the  data,  collected  throughout  the  life-cycle 
of  the  SSTF,  will  verify  that  the  objectives 
are  being  met. 

The  Author  Environment  will  contain  the 
following  systems.  Each  system  will  include 
hardware,  software,  and  procedures  which 
may  be  shared  with  other  systems. 

0  Development 
0  Reconfiguration 
o  Operations  and  Maintenance 
0  Test  and  Verification 
0  Management  Information  Center 
o  Product  Consolidation  and  Distribution 
0  Visual  Scene  Generation 
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Development.  Development  produces  new  or 
modified  SSTF  capabilities  for  both  the 
Author  Environment  and  the  Student  Envi¬ 
ronment.  All  SSTF  systems  ranging  from  the 
real-time  simulation  to  institutional  pro¬ 
cesses  are  eligible  candidates  for  Develop¬ 
ment  attention. 

A  Software  Support  Environment  (SSE)  is 
presently  being  built  to  support  the  devel¬ 
opment  of  all  operational  SSFP  software. 
This  SSE  will  provide  an  Ada  Programming 
Support  Environment  (APSE)  including 
software  life-cycle  tools,  rules,  and  stan¬ 
dards,  and  a  Common  APSE  Interface  Set 
(CAIS)  for  the  SSTF  computer  hardware. 
The  SSE  will  be  implemented  on  multiple 
facilities,  called  Software  Production  Facili¬ 
ties,  one  of  which  will  serve  as  the  Develop¬ 
ment  System  of  the  SSTF. 

Creation  of  new  SSTF  systems  and/or  sub¬ 
systems  may  also  require  hardware  or  pro¬ 
cedures  design  and  development.  These 
activities  are  the  responsibility  of  Develop¬ 
ment  and  may  be  conducted  at  the  SSTF  or  at 
other  facilities. 

Development  will  support  the  SSTF  goals  of 
high  productivity,  low  life-cycle  cost, 
reliability,  maintainabiiity,  and  operability. 
Development  will  also  support  the  SSFP 
objectives  of  Automated  Systems  and  Arti¬ 
ficial  Intelligence  advancement. 

The  following  activities  must  be  supported 
by  Development: 

o  Coordination  of  all  simulation  and 
training  requirements  from  all  the 
SSFP  participants. 

o  Providing  all  support  required  for  the 
Simulation  Facility  Director, 
o  Providing  assistance  with  problem 
diagnosis  and  repair  for  both  hardware 
and  software  problems. 

0  SSTF  planning  and  scheduling. 

The  users  of  the  Author  Environment  require 
the  full  spectrum  of  software  development 
capabilities  afforded  by  the  SPF.  Develop¬ 
ment  users  require,  at  a  minimum,  the 


capabilities  to  edit  copies  of  existing  source 
code,  create  new  source  code,  compile  source 
code,  debug  the  code,  and  eventually  execute 
the  code  in  a  training-like  system  separated 
from  the  Student  Environment.  The  users 
will  also  require  access  to  individual  copies 
of  all  or  parts  of  the  SSTF  training  and  sim¬ 
ulations  software  to  assist  in  the  develop¬ 
ment  and  testing  of  new  systems.  This  means 
that  users  require  tools  that  allow  the  iden¬ 
tification  of  the  required  software  modules 
and  the  ability  to  transfer  them  among  SSTF 
systems. 

Development  users  require  access  to  com¬ 
puter  resources,  configuration  management 
information,  and  software  development  and 
testing  tools.  Furthermore,  access  to  these 
various  resources  must  be  established  via  a 
common  interface,  thereby  allowing  all  of  a 
user's  work  to  be  performed  from  a  single 
terminal  or  workstation. 

Reconfiouration.  Reconfiguration  provides 
the  capabilities  necessary  to  integrate 
existing  and  newly-developed  real-time 
software  into  Student  Environment  training 
loads  to  meet  the  changing  Station  con¬ 
figuration-specific  requirements.  The 
resulting  training  loads  reflect  new  Station 
configuration-specific  training  scenario.  As 
the  Product  Consolidation  and  Distribution 
archive  of  training  load  components 
increases  in  number.  Reconfiguration  pro¬ 
vides  the  majority  of  the  training  loads  from 
existing  load  components  rather  than  from 
newly  developed  load  components. 

Because  of  the  changing  nature  of  the  Sta¬ 
tion's  configuration  and  its  projected  long 
life  span.  Reconfiguration  will  automate  the 
assembly  of  training  loads  rather  than  using 
the  historical  method  of  custom  designing  and 
hand  fitting  of  the  software  programs  that 
comprise  a  training  load. 

Operations  and  Maintenance.  Operations 
and  Maintenance  will  comprise  hardware, 
software,  procedures,  and  user  services  that 
support  the  daily  operations  and  problem 
identification,  investigation,  evaluation,  and 
reporting  for  all  of  the  SSTF. 
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Services  provided  by  Operations  and  Main¬ 
tenance  include 

0  Audit  Trail  of  System(s)  Usage 
0  Collection  of  Training  Effectiveness 
Data 

0  Configuration  Management 
o  Hardware  Maintenance 
0  Hardware  Modification 
0  Libraries  of  Documents  and  Software 
0  Securing  Files 

o  Training  for  Operations  Personnel 

In  addition  to  these  services,  Operations  and 
Maintenance  will  provide  the  following  ser¬ 
vices  specifically  for  the  Author  Environ¬ 
ment: 

o  Problem  Incident  Reports 
0  Scheduling 
o  Help  Desk  Staffing 

Tpfit  and  Verification.  Test  and  Verification 
will  contain  test  beds  for  the  hardware  and 
software  that  goes  into  the  Student  Environ¬ 
ment  and  will  contain  hardware  test  equip¬ 
ment  such  as  logic  analyzers,  and  software 
quality  assurance  tools.  Tes.  and  Verifica¬ 
tion  has  two  major  objectives:  to  ensure  the 
ability  of  the  subsystem  elements  to  operate 
together  as  an  integrated  system  and  with 
other  systems,  and  to  evaluate  system  and 
subsystem  performance  and  compliance  with 
requirements. 

Management  Information  Center.  The  Man¬ 
agement  Information  Center  will  provide  the 
ability  to  collect,  collate,  distribute,  and 
archive  all  management  information  perti¬ 
nent  to  the  daily  operations  of  the  SSTF.  It 
interfaces  with  other  SSFP  information 
centers  for  the  efficient  transfer  of  data 
between  SSFP  facilities.  The  Management 
Information  Center  will  include  the  follow¬ 
ing  capabilities: 

0  Computer  performance  management 
0  Configuration  management 
0  Data  analysis  and  graphing 
0  Discrepancy  reporting 
0  Electronic  mail 
0  Facility  scheduling 
o  Facility  user  identification 


0  Logistics  management 
o  Organizational  structure  and 
directory  information 
o  Contractor  task  authorization 
0  Simulation  Facility  Director's 
daily  notes 

o  Simulation  task  status  information 
o  Task  authorization  and  tracking 
0  Technical  reports  and  product 
documentation 

0  Test  scenarios,  results,  and  reports 
o  Training  records 

Product  Consolidation  and  Distribution. 
Product  Consolidation  and  Distribution  will 
be  a  software  repository  for  all  SSTF  train¬ 
ing  module  components.  It  will  be  used  to 
support  training  load  building  and  to  archive 
software  modules.  Completed  training  loads 
modules,  created  by  Development  or  con¬ 
structed  by  Reconfiguration,  will  be 
archived  here.  All  software  modules  stored 
in  Product  Consolidation  and  Distribution 
are  under  configuration  control. 

Visual  Scene  Generation.  Visual  Scene  Gen¬ 
eration  will  be  used  to  develop  and  test 
visual  scenes  for  use  in  the  Student  Envi¬ 
ronment.  The  data  created  by  Visual  Scene 
Generation  will  be  archived  in  Product  Con¬ 
solidation  and  Distribution. 

STUDEIVT  ENVIRONMENT 

The  role  of  the  Student  Environment  will  be 
to  support  the  presentation  of  training  to 
SSFP  students.  The  Student  Environment 
contains  the  trainers  that  are  used  to  train 
Station  astronaut  crew  members,  ground 
support  personnel,  and  the  JSC  Training 
Division  instructors  for  generic  and  Station 
configuration-specific  activities.  In  addition 
to  the  trainers,  the  Student  Environment 
contains  support  equipment  and  subsystems 
that  are  necessary  for  training  operations. 

The  Student  Environment  uses  hardware  and 
software  to  perform  simulations,  and  auto¬ 
mated  or  manual  procedures  that  guide  the 
use  of  the  software  to  accomplish  the 
required  training.  The  following  objectives 
are  used  as  the  guidelines  for  designing  and 
operating  the  Student  Environment: 
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o  Satisfy  evolving  training  needs 
and  requirements, 
o  Provide  adequate  training  tasks 
and  facilities. 

o  Use  effective  training  methodologies. 

TRAINERS 

The  Student  Environment  will  contain  a  va¬ 
riety  of  trainers  which  represent  compo¬ 
nents  of  the  Space  Station.  Figure  1  shows  a 
proposed  configuration  of  trainers  in  the 
SSTF.  These  trainers  are  described  below. 

Module  Systems  Trainer.  The  Module  Sys¬ 
tems  Trainers  (MSTs)  will  be  used  to  train 
astronauts  in  operations  performed  at  the 
Multi-Purpose  Applications  Consoles 
(MPACs)  that  will  be  on-board  the  Station 
[3].  There  will  be  four  MSTs  in  the  SSTF 
corresponding  to  the  four  modules  of  the 
Space  Station  [4]:  the  Habitation  (HAS) 
Module,  the  Laboratory  (LAB)  Module,  the 
Japanese  Experiment  Module  (JEM),  and  the 
European  Space  Agency  (ESA)  Module.  The 
MSTs  may  be  operated  independently  or  in 
combination  with  other  Student  Environment 
training  systems.  The  MST  will  support  the 
following  training: 

o  Activity  Planning 
0  Communications  Usage 
o  Emergencies  Simulation 
0  Maintenance  and  Logistics 
0  Nominal  and  Contingency  Systems 
Management 

0  Trajectory  and  Navigation  Support 
o  Use  of  Onboard  Training  Facilities 

Station  Proximity  Operations  Trainer.  The 
Station  Proximity  Operations  Trainer 
(SPOT)  can  be  used  as  either  an  augmented 
MST  or  as  a  representation  of  the  working 
environment  of  a  Station  node  with  an  oper¬ 
able  cupola  represented  by  the  visual  system 
of  the  SPOT.  The  visual  system  surrounds 
the  SPOT  cupola  and  is  used  to  present  visual 
images  necessary  for  the  development  of 
crew  member  skills  requiring  hand-to-eye 
coordination  or  precise  control.  The 
manipulative  skills  learned  in  the  SPOT  will 
be  applied  to  the  proximity  operations  that 


are  accomplished  from  within  the  Station 
such  as  attached  payload  servicing,  external 
Station  systems  maintenance,  logistics  mod¬ 
ule  exchange,  extravehicular  activity  (EVA) 
support.  Orbital  Maneuvering  Vehicle 
berthing  and  release,  and  man-tended  free- 
flyer  berthing  and  release. 

Node  Systems  Trainer.  The  Node  Systems 
Trainer  will  be  used  to  train  crew  members 
for  tasks  that  are  performed  in  the  nodes  of 
the  Space  Station,  including  tasks  that 
require  the  use  of  an  MPAC. 

Ground  Systems  Trainer.  The  Ground  Sys¬ 
tems  Trainer  (GST)  will  be  used  to  train 
Space  Station  Control  Center  (SSCC)  ground 
support  personnel  in  SSCC  workstation 
activities.  The  training  includes  normal  and 
emergency  procedures  required  to  support 
Station  operations.  The  GST  contains  soft¬ 
ware.  hardware  controls,  and  displays  that 
accurately  reproduce  both  the  function  and 
appearance  of  the  workstations  located  in  the 
^CC.  Each  GST  has  voice,  command,  and  data 
communications  with  all  Student  Environ¬ 
ment  trainers. 

Computer  Assisted  Instructional  Trainer. 
The  Computer  Assisted  Instructional  Trainer 
(CAIT)  provides  single  user,  introductory  or 
refresher  training.  The  CAIT  is  not  con¬ 
nected  to  the  other  trainers  and  does  not 
participate  in  SSTF  real-time  training.  The 
CAIT  contains  four  student  workstations 
connected  to  a  dedicated  lesson  development, 
testing,  storage,  and  distribution  system. 
The  CAIT  is  connected  to  a  network  for  voice 
communications  and  for  file  transfer. 

SUPPORTING  SYSTEMS 

A  number  of  SSTF  supporting  systems  will 
be  located  in  the  Student  Environment  to 
support  the  operation  of  the  trainers  and  to 
interface  the  trainers  to  other  SSFP  sys¬ 
tems. 

Instructor  Station.  An  Instructor  Station 
(IS)  will  be  used  to  monitor  and  control  each 
trainer.  Examples  of  monitor  and  control 
capabilities  include  mode  control,  malfunc¬ 
tion  insertion,  and  data  retrieval.  An 
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instructor  will  use  these  capabilities  to 
conduct  a  training  session;  operations  and 
maintenance  engineers  will  use  them  to 
diagnose  simulation  failures;  and  training 
load  developers  will  use  them  to  integrate 
and  test  new  simulation  capabilities. 

Each  Instructor  Station  will  have  communi¬ 
cations  with  other  Instructor  Stations,  the 
SSTF  trainer  currently  controlled  by  the 
Instructor  Station,  the  SSCC,  and  the  Mission 
Control  Center.  At  a  minimum,  each 
Instructor  Station  will  support  the  following 
capabilities: 

o  Starting  and  terminating  a  training 
session 

o  Control  of  the  simulations  and  data 
logging  and/or  delogging 
0  Support  of  external  interfaces 
o  Instructor-to-student  communications 
0  Malfunction  insertion,  removal  and 
activation 

o  Training  report  generation 
o  Simulator  performance  monitoring  via 
suitable  graphics  and/or  alphanumeric 
displays 

The  Instructor  Station  also  allows  instruc¬ 
tors  to  control  the  simulation  with  moding 
commands  such  as  [5]: 

o  Freeze 
o  Run 
o  Reset 
0  Data  store 
o  Reconfiguration 
o  Pre-Run 
0  Step  Ahead 
o  Variable  Time 
0  Safe  Store 

Simulation  Facility  Director  Console.  The 
Simulation  Facility  Director  (SFD)  Console 
will  allow  monitoring  and  control  of  the 
daily  operations  of  the  SSTF,  Any  Instructor 
Station,  when  accessed  with  the  proper  sign- 
on  codes,  can  be  used  as  the  SFD  console. 
Normally,  two  Instructor  Station  units  will 
be  located  in  the  designated  SFD  work  area 
and  will  be  dedicated  for  use  as  the  SFD  con¬ 
sole.  The  SFD  console  has  communications 
with  all  SSTF  systems,  the  SSCC,  the  entire 


SSFP  communication  network,  and  NASA 
institutional  systems. 

The  SFD  will  maintain  active  control  of  the 
Author  Environment  and  the  Student  Envi¬ 
ronment,  schedule  access  to  SSTF  facilities 
and  systems,  schedule  maintenance  person¬ 
nel,  prepare  SSTF  utilization  reports,  co¬ 
ordinate  acceptance  of  development  deliver¬ 
ables,  coordinate  reconfiguration,  and  assign 
change  request  or  discrepancy  report 
responsibility. 

SSIS  Network  Simulator.  The  Space  Station 
Information  System  (SSIS)  Network  Simu¬ 
lator  (SNS)  will  model  the  SSIS  ground  seg¬ 
ment  [6]  and  will  have  the  ability  to  inter¬ 
face  the  SSTF  to  the  SSCC  via  the  real  world 
SSIS.  The  SNS  will  use  the  SSIS  to  connect 
SSTF  trainers  (except  the  CAIT)  to  SSFP 
facilities  that  are  external  to  the  SSTF  (for 
example,  the  SSCC  or  the  Payload  Operations 
Integration  Center).  The  SNS  simulation 
models  include  the  Data  Interface  Facility 
(DIF),  the  Tracking  and  Data  Relay  Satellite 
System  (TDRSS),  and  portions  of  the  SSIS 
that  are  located  on  Space  Station  Freedom. 

In  addition  to  uplink,  downlink,  multiplexer 
and  demultiplexer  simulation,  the  SNS  will 
simulate  the  SSIS  Tracking  System,  the  SSIS 
Network  Integration  Management,  and  the 
SSIS  Resource  Scheduling  System.  Implicit 
in  this  is  the  necessity  to  simulate  the 
TDRSS  Network  Systems  and  all  SSIS  and 
TDRSS  protocols  involved  in  transmitting, 
receiving,  acknowledging,  validating,  and 
executing  network  commands. 

TRAINING  MODES 

To  support  the  wide  variety  of  training 
requirements,  the  trainers  in  the  SSTF 
(except  for  the  CAIT)  will  operate  in  the 
following  five  modes  [5]. 

Standalone.  A  standalone  training  session  is 
one  which  utilizes  a  single  trainer  to 
accomplish  the  training  objective(s). 

Combined.  A  training  session  is  classified 
as  combined  if  any  combination  (two  or 
more)  of  the  SSTF  trainers  participate  in  a 
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third  session  (represented  by  the  thin  solid 
line)  is  a  combined  session  consisting  of  a 
MST,  the  SNS  and  one  GST.  The  fourth  ses¬ 
sion  (represented  by  the  thick  dotted  line)  is 
a  combined  session  consisting  of  the  SNS  and 
the  other  GST.  The  fifth  session  is  a  stand¬ 
alone  NST. 
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session  to  accomplish  the  training  objec- 
tive(s). 

Joint-Combined.  A  training  session  is 

classified  as  joint-combined  if  it  involves 
one  or  more  SSTF  trainers  plus  at  least  one 
facility  outside  the  SSTF  other  than  the  SSCC. 

Integrated.  Training  is  classified  as  inte¬ 
grated  when  any  of  the  SSTF  trainers  par¬ 
ticipate  in  a  training  session  with  the  SSCC. 

Joint-Intearated.  A  training  session  is 

classified  as  joint-integrated  if  it  is  an 
integrated  training  session  that  also  includes 
participation  of  a  facility  external  to  JSC. 

Figure  2  illustrates  joint-combined  training 
sessions.  The  trainers  in  the  SSTF,  along 
with  external  facilities,  are  participating  in 
five  training  sessions.  The  Shuttle  Mission 
Training  Facility  (SMTF),  the  Space  Station 
Mockup  and  Trr’ner  Facility  (SSMTF)  and 
the  Neutral  Buoyancy  Laboratory  (NBL)  at 
JSC  are  connected  to  the  SSTF  in  this  sce¬ 
nario.  In  addition,  the  Payload  Operations 
Training  Facility  (POTF)  at  Goddard  Space 
Flight  Center  (GSFC)  and  the  Integration, 
Test,  and  Verification  Facility  (ITVF)  at 
Marshall  Space  Flight  Center  (MSFC)  are 
participating  in  joint-combined  training 
sessions. 

All  five  typ^r  of  training  sessions  are 
included  in  tiie  set  of  sample  training  sce¬ 
narios  illustrated  in  Figures  3,  4  and  5. 

Figure  3  shows  a  scenario  with  five  con¬ 
current  training  sessions.  The  first  training 
session  (represented  by  the  thin  solid  line) 
is  an  integrated  simulation  coupling  the 
SPOT  with  the  SSCC.  For  example,  the 
astronaut  in  the  SPOT  may  be  practicing 
grappling  a  payload  with  the  Mobile  Remote 
Manipulator  System  (the  manipulator  arm) 
while  coordinating  the  procedure  with  the 
flight  controllers  ir.  the  SSCC.  In  this  ses¬ 
sion  the  SNS  is  simulating  the  data  flow 
between  Space  Station  Freedom  and  the  SSCC 
via  the  TDRSS  and  the  DIF.  The  second 
traininr  session  (represented  by  the  dashed 
line)  is  a  combined  session  consisting  of  one 
MST  connected  to  a  GST.  This  could  be  an 


astronaut  practicing  environmental  control 
and  life  support  procedures  while  flight 
controllers  train  on  monitoring  those  sys¬ 
tems  on  the  ground.  The  SNS  is  participating 
in  this  session,  too.  The  third  session  is  a 
stand-alone  MST.  This  could  be  an  astronaut 
becoming  familiar  with  the  operation  of  the 
electrical  power  system  of  Space  Station 
Freedom.  The  fourth  session  (represented 
by  the  dcned  line)  is  a  joint-combined  ses¬ 
sion  with  the  NBL.  The  astronaut  in  the  NBL 
is  training  for  EVA  and  communicates  with 
an  astronaut  in  the  MST  to  practice  the 
coordination  of  activities  that  will  be 
required  on-orbit.  The  fifth  session 
(represented  by  the  thick  solid  line),  like 
the  second  session,  is  a  combined  session.  It 
has  an  MST  connected  to  a  GST  through  the 
SNS.  The  two  NSTs  are  inactive  in  this  sce¬ 
nario. 

Figure  4  shows  a  scenario  which  has  five 
concurrent  training  sessions,  the  first  of 
which  is  a  joint-combined  session  with  the 
SMTF.  This  session  includes  the  SPOT  and 
the  SNS  in  the  SSTF  and  it  includes  the  Fixed 
Base  Simulator  (a  Space  Shuttle  simulator) 
and  the  Network  Systems  Simulator  (NSS) 
in  the  SMTF.  In  this  session  the  visual  sys¬ 
tems  of  the  SMTF  and  the  SSTF  are  synchro¬ 
nized  to  practice  coordination  of  manipulator 
arm  operations  between  the  Space  Shuttle 
and  Space  Station  F reedom.  An  example  of 
such  an  activity  is  a  payload  being  taken  out 
of  the  Shuttle  payload  bay  by  the  Shuttle's 
manipulator  arm  and  then  being  grappled  by 
a  Space  Station  Freedom  manipulator  arm 
for  stowage.  The  remaining  four  sessions  in 
this  sce'iario  are  stand-alone  sessions.  In 
these  session  four  astronauts,  two  in  MSTs 
and  two  in  NSTs,  are  undergoing  solo  train¬ 
ing. 

Figure  5  depicts  another  scenario  with  five 
concurrent  training  sessions.  In  the  first 
session  (represented  by  the  thick  solid 
line),  the  SPOT  and  two  MSTs  are  perform¬ 
ing  a  joint-integrated  session  with  the  SSCC 
and  the  Payload  Operations  Integration  Cen¬ 
ter  at  Marshall  Space  Flight  Center.  Since 
this  session  simulates  communications 
between  Space  Station  Freedom  and  ground 
facilities,  the  SNS  is  participating  in  it.  The 
second  session  is  a  stand-alone  MST.  The 
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Figure  3:  Five  Concurrent  Training  Sessions 
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Figure  4:  Five  Concurrent  Training  Sessions 


AN  INSTRUCTOR  COHMDNICATION  FRAMEVORR  FOR  TRAINING  SIMULATORS 


Hilbert  Kuiper,  Geert  F.  Slegtenhorst,  Rob  den  Heijer 


This  paper  describes  an  Instructor  Commtinication  Module  (ICM) 
that  is  part  of  a  Universal  Computer  Assisted  Instruction  (CAI) 

System  taking  care  of  the  automatic  training  process  in  real-time 
training  simulators,  such  as,  for  Instance,  tank-  or  flight  simulator 
for  operator  training.  The  system  is  based  upon  a  tailor-made 
training  system  that  was  developed  some  years  ago  under  the 
supervision  of  the  TNO  Physics  and  Electronics  Laboratory. 

The  ICM  supports  the  tasks  of  the  instructor.  To  determine  what 
tasks  are  necessary,  a  task  analysis  has  been  carried  out  by 
generalizing  the  tasks  in  an  existing  training  simulator.  As  a 
result,  five  main  instructor- tasks  can  be  distinguished:  System 
Management,  Result  Overview,  Student  Progress,  Judgement  and 
Briefing. 

The  main  goal  was  to  make  the  ICM  universal  in  two  ways:  firstly, 
applicable  for  several  training  simulators  and  secondly,  workstation 
independent . 

A  feasibility  model  has  been  developed  using  the  programming 
language  C  and  the  X-window  system  on  a  commercially  available 
workstation.  The  model  has  one  main  window  for  the  global  overview 
and  other  windows  can  be  opened  optionally.  Direct  manipulation  and 
object  oriented  techniques  have  been  implemented. 

The  first  evaluation  results  of  this  model  are  positive,  but  give 
also  rise  to  some  adjustments  of  the  user- interface. 

Future  enhancements  include:  conversion  to  the  programming 
language  Ada,  extension  to  a  full  prototype  and  adding  on-line  help 
facilities. 
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Figure  5:  Five  Concurrent  Training  Sessions 


AN  INSTRUCTOR  COMMUNICATION  FRAMEVORK  FOR  TRAINING  SIMULATORS 


Hilbert  Kulper,  Geert  F.  Slegtenhorst,  Rob  den  Heljer 


Introduction  In  a  training  situation,  such  as,  for  instance,  a 
simulator  configuration  for  training  a  flight  crew  or  a  tank  crew, 
complex  and  time -critical  processes  have  often  to  be  monitored  and 
judged  for  several  simulators  at  the  same  time.  The  application  of 
Computer  Assisted  Instruction  (CAI)  in  such  a  situation  is  highly 
recommended . 

Only  in  this  manner,  it  is  possible  to  record  what  the  student  is 
doing  during  the  training -process  and  to  give  an  objective  Judgement 
that  is  detailed  enough. 

The  Netherlands  anti-aircraft  tank,  the  35  mm  PRTL  (Pantser  Rups 
Tegen  Luchtdoelen) ,  is  an  example  of  a  system  where  time-critical 
processes  are  Important.  A  training  system  for  this  tank  has  been  in 
use  for  40  hours  a  week,  since  1980. 

This  training -system,  the  PLT  (Pantser  Lua  Trainer) ,  was  developed 
under  the  supervision  of  the  TNO  Physics  and  Electronics  Laboratory. 
In  the  PLT  tailor-made  CAI  has  been  used  in  depth.  The  objective  of 
the  system  is  the  training  of  the  crew  of  two  persons  in  operating 
the  tank. 

The  crew  is  housed  in  the  simulator,  a  feel  and  look-alike  copy  of 
the  original  tank- turret.  The  PLT  consists  of  three  simulators  and  is 
able  to  train  three  crews  of  two  persons  each  simultaneously  and 
independently.  In  general  we  will  call  these  simulators  or  turrets 
learning  stations. 

An  instructor  can  monitor  the  training-processes  in  the  active 
turrets  through  an  instructor  console  simultaneously.  In  this  way  he 
.rets  an  overview  of  the  actions  and  the  progress  of  the  crews  being 
txdxned. 

A  training  system  can  handle  one  or  more  learning  stations 
Independently  of  each  other  or  together  as  a  team. 

When  operating  in  the  independent  mode,  all  learning  stations  can 
follow  different  scenarios;  when  operating  as  a  team  they  all  have 
the  same  scenario. 


Figure  1  Trainer  configuration 
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Figure  1  shows  the  training  situation  schematically,  in  which  the 
learning  stations  or  simulators  are  controlled  from  the  instructor 
station  by  the  CAI -process  or  by  the  instructor. 

The  CAI -process  runs  on  a  computer  that  is  integrated  in  the 
Instructor  console.  The  instructor  has  access  to  the  system  by  means 
of  a  workstation  or,  for  Instance,  a  console  terminal. 

The  training  of  the  crew  in  a  learning  station  is  divided  into 
several  training  sessions.  A  training  session  is  a  certain  time- 
period  (for  Instance  one  hour)  in  which  the  crew  receives  a  number  of 
lesson  modules.  Which  and  how  many  lesson  modules  are  followed  by  the 
crew  depend  on  their  results. 

The  Universal  CAI  System  At  the  TNO  Physics  and  Electronics 
Laboratory,  a  CAI  System  is  currently  being  developed,  based  upon  the 
CAI  in  the  PLT.  This  system  will  be  universally  applicable  for  a 
broad  range  of  simulators.  The  Universal  CAI  System  (UCS)  has  the 
advantage  that  only  a  single  development  of  the  CAI  Sytem  is 
necessary  and  that  only  the  simulator  and  the  interface  to  the  UCS 
have  to  be  developed  for  a  new  application.  The  basic  components  of 
the  UCS  are  depicted  in  figure  2. 
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Figure  2  The  Universal  CAI  system 


System  definition 

With  this  component,  the  target  simulator  is  specified.  All 
states,  switches  etc.  within  the  simulator  that  are  relevant  for 
the  training,  are  identified  and  get  a  function  name. 

CAI  preparation 

The  preparation  component  is  responsible  for  the  creation  of  the 
lesson  modules,  in  which  the  scenarios,  judging  criteria  and  feed¬ 
back  messages  are  defined. 
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Universal  CAI  Module  (UCM) 

This  part  comprises,  in  fact,  the  real-time  training.  This 
includes  the  presentation  of  the  scenarios  to  the  simulator,  the 
analysis  of  the  recorded  student  actions,  the  judgement  of  the 
results  and  the  determination  of  the  lesson  progress  (the  next 
scenario).  This  part  is  comparable  to  the  CAI-process  in  figure  1. 

Student  administration 

The  data  of  the  students  are  stored  and  analysed.  These  data  are 
related  to  the  history  of  the  student,  his  results,  his  weaknesses 
etc . 

At  this  time,  only  the  real-time  training  component,  the  Universal 
CAI  Module,  has  been  prototyped. 

The  Universal  CAI  Module  (UCM)  The  UCM,  responsible  for  the  real¬ 
time  training,  has  an  instructional  part  and  a  number  of  modules 
taking  care  of  the  communication  with  the  other  parts  of  the  trainer. 
The  larger  part  of  the  UCM  is  platform  independent.  In  figure  3,  the 
components  of  the  UCM  are  depicted  within  an  overall  ICAI- 
architecture  (ICAI-Intelligent  Computer  Assisted  Instruction) . 


Preparation 


Figure  3  UCM-ICAI  Archltecttire 


The  main  activities  of  the  instructional  process  are  depicted 
schematically  in  figure  4. 
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Figure  4  Stages  of  Instruction  process 

At  the  start  of  a  session  the  progress  component  determines  the 
proper  lesson  module  for  the  students,  based  upon  their  results. 

After  setting  up  this  module,  the  simulation  starts  and  all  relevant 
student  actions  are  recorded.  At  the  end  of  the  simulation  (lasting  3 
till  5  minutes)  the  recorded  data  are  gathered  by  the  analysis 
component  and  compared  with  the  ideal  situation.  A  judgement  is  made 
up,  a  score  is  determined  and  the  process  starts  again  with  a  new 
lesson  module  determined  by  the  progress  component. 

In  certain  cases,  the  progress  determination  can  also  be  done 
manually,  by  the  instructor,  instead  of  the  instructional  process. 
Scenario  and  judging  criteria  constitute,  among  others,  a  so-called 
lesson  module,  a  certain  time  of  simulation.  A  lesson  module  can  have 
different  shapes.  Normally,  a  lesson  module  is  part  of  a  chain  of 
modules,  each  presenting  a  new  piece  of  training  material.  These 
modules  are  called  headline-modules.  A  lesson  module  can  also  be  a 
repetition  of  the  former  headline -module;  in  this  case  it  is  called  a 
repetitive  module.  Finally,  a  lesson  module  can  also  focus  on  a 
certain  aspect  of  a  headl Ine -module ,  in  which  case  it  is  a  corrective 
lesson  module. 

From  figure  3  it  is  clear  that  the  instructional  process  is 
communicating  with  a  number  of  modules; 

Simulator  communication  module 

This  module  receives  the  recorded  actions  of  the  student  from  the 
simulator. 

Preparation  module 

This  module  delivers  the  prepared  library  of  lesson  modules  and 
the  lesson  network. 
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Administration  module 

This  module  takes  care  of  updating  the  student's  personal  data 
based  upon  his  results  during  the  training. 

Student  communication  module 

Through  this  module  the  student  receives  feedback  inside  the 
simulator  and  his  response  is  processed. 

Instructor  communication  module 

With  this  module,  the  input  from  the  instructor  is  processed  and 
he  gets  an  overview  of  the  progress  in  every  session. 


The  remainder  of  this  paper  will  focus  on  the  instructor 
communication  module. 


The  Instructor  Communication  Module  The  purpose  of  the  Instructor 
Communication  Module  (ICM)  is  the  support  of  the  instructor's  tasks. 
This  means  that  the  instructor  must  have  the  facility  to  monitor  all 
the  training  sessions  at  the  same  time,  wherever  he  thinks  it  is 
necessary.  To  be  able  to  determine  exactly  what  to  support,  the  tasks 
of  the  Instructor  are  specified  in  detail. 

A  global  partition  of  the  supported  tasks  is: 

-  system  management 

-  result  overview 

-  student  progress 

-  judgement 

-  briefing 

The  tasks  This  paragraph  gives  a  more  detailed  description  of  the 
aforementioned  tasks. 

System  Management. 

As  the  administrator  of  the  training  system,  the  instructor  has  to 
start  up  the  system  and,  at  the  end  of  all  training  sessions,  must 
close  it  down.  Also,  he  has  to  start  up  the  different  sessions.  He 
has  to  indicate  which  learning  stations  are  training  together, 
which  students  are  in  these  learning  stations,  with  which  scenario 
the  session  is  going  to  start  with  and  at  what  time  the  session 
will  be  ending.  Besides,  the  instructor  sometimes  has  to  abort  or 
end  sessions  at  an  earlier  time  than  determined  by  the  system. 

Result  Overview. 

The  instructor  has  to  inspect  results,  should  have  an  idea  what 
the  students  are  doing  in  their  learning  stations.  This  could  be 
globally  or  in  detail.  Global  inspection  means  that  the  instructor 
can  see  at  glance  what  sessions  are  active,  what  learning  stations 
are  in  that  session,  what  lesson  module  is  active  in  the  session 
etc. 

Detailed  inspection  entails  that  the  instructor  can  monitor  the 
real'time  simulation  with  specific  screens  in  one  of  the  learning 
stations,  but  also  that  information  can  be  retrieved  concerning: 
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-  results:  what  went  wrong  at  a  certain  lesson  module  and  why, 

•  progress:  what  lesson  modules  were  in  the  session  and  in  what 
order , 

-  personal  data:  student  identification  and  his  former  results. 
Student  Progress. 

A  situation  in  which  the  instructor  will  decide  to  intervene  in 
the  active  training  process  is  conceivable.  This  implies  that  not 
the  instructional  process,  but  the  instructor  will  determine  the 
next  lesson  module .  The  manual  determination  of  the  progress  may 
be  necessaiTT  if  the  student  continuously  gets  bad  results.  The 
student  can  also  ask  for  an  intervention  of  the  instructor. 
Finally,  the  Instructor  himself  can  decide  that  manual  progress 
determination  must  take  place.  Determining  the  next  lesson  module, 
the  Instructor  can  make  a  choice  from  one  of  the  following  items: 

•  a  headline -module ; 

-  skipping  other  headline -modules; 

-  a  corrective  lesson  module;  deals  with  a  certain  aspect  of  the 
headline -module ; 

-  a  repetitive  module,  a  repetition  of  the  headline -module 
possibly  with  other  judgement  criteria. 

Judgement . 

The  instructor  may  have  the  opinion  that  the  judgement  of  a  lesson 
module  by  the  instructional  process  is  not  completely  correct.  He 
can  adjust  this  judgement  himself.  If  he  feels  that  the  judgement 
was  a  correct  representation,  but  that  this  was  more  the  result  of 
the  individual  effort  of  a  single  student  rather  than  the  pair  of 
them,  he  can  change  the  score  allocation.  In  both  cases,  he  also 
acts  as  a  corrector  of  the  instructional  process. 

The  instructor  can  also  enter  new  scores  for  certain  aspects  of 
the  training  about  which  the  instructional  process  is  unable  to 
judge,  like  for  instance,  the  oral  communication  protocol.  In  this 
case,  the  instructor  is  supplementary  to  the  instructional 
process . 

Briefing. 

Of  course,  the  instructor  also  has  to  perform  Instructive  tasks 
during  the  training.  This  mainly  takes  place  by  means  of  oral 
communication  with  the  student  with  the  aid  of  an  intercom  or 
microphone.  Besides  the  oral  communication,  the  instructor  can 
also  send  text  messages  to  the  learning  stations. 

Furthermore,  the  instructor  can  adapt  values  in  certain  registers 
in  a  learning  station,  like  for  instance,  the  fill-up  of  fuel  for 
a  learning  station  used  for  driving  simulation,  when  the  students 
have  used  all  the  fuel. 

Task  support  is  activated  by  the  reaction  of  the  module  to  the 
instructor's  wishes.  The  module  can  take  care  of  both  global  and 
detailed  overviews,  b-^longing  to  its  overview  task.  It  also  controls 
the  interface  function  between  the  Instructor's  wishes  and  the 
reaction  of  the  instructional  process.  It  is  clear  that  the 
development  of  the  user  interface  of  this  module  requires  much 
attention  because  of  its  important  role. 
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Designing  the  user  Interface  A  user  interface  specifies  how  the 
system  is  presenting  itself  to  the  user;  the  human - computer 
interaction  is  stated  in  this  user - interface .  The  importance  of  an 
extended  design  of  a  user- interface  in  an  early  stage  of  the  design 
process  is  being  recognized  more  and  more.  From  a  functional  point  of 
view  the  instructor  module  user- interface  is  an  important  part. 

During  the  design  of  the  instructor  module  a  graphical  window  system 
was  chosen  as  an  environment  for  the  user  Interface.  Advantages  of 
this  choice  are  that  the  initiative  and  the  overview/monitoring  lie 
with  the  user  and  that  the  computer  memory  usage  is  low  when  compared 
to  a  command  language  These  user -interfaces  also  give  the  users  a 
satisfactory  feeling. 

In  a  user- interface  based  upon  windows,  the  screen  is  divided  into 
windows;  these  are  functional  areas  selected  by  the  user  which  can  be 
sometimes  manipulated. 

Within  the  instructor  module  a  main  window  is  functlonli.g  as  a 
central  functional  area,  giving  a  global  overview  of  all  current 
training  sessions.  Other  instructor-facilities  can  be  chosen  by  means 
of  a  menu.  In  figure  S  we  see  a  hard-copy  of  this  main  window. 
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Figure  5  Main  menu  from  the  Instructormodule 

In  the  global  overview  we  see  which  training  session  is  active,  which 
learning  stations  within  a  session  are  training  as  a  team,  which 
lesson  module  is  active  for  them,  how  far  the  simulation  of  the 
lesson  module  has  progressed,  which  students  are  in  the  learning 
stations  etr . 

The  menubuttons  in  the  menubar  under  the  global  overview  are  related 
to  the  respective  tasks  of  the  instructor. 

Whereas  the  main  window  is  always  visible,  temporary  windows  can  be 
opened  with  the  aid  of  menu  choices.  These  could  be  input  for 
commands  screens  or  screens  presenting  overviews. 

With  the  aid  of  the  input  screen  the  instructor  can  specify  what  he 
wants  to  pass  through  to  the  Instructional  process,  such  as  the 
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determination  of  the  progress  or  the  change  of  a  i-.ore.  In  figure  6 
we  see  an  example  of  an  input  screen,  in  which  the  instructor  can 
determine  what  progress  he  ""hinks  is  necessary  for  a  certain  session 


Figure  6  Commaud  inputscreen  -  progress  determination 

An  overview  screen  gives  detailed  irformation  about  lesson  module 
results,  session  progress  or  personal  data.  In  figure  7  we  see  a 
screen  with  an  overview  of  the  session  progress.  This  overview 
consists  mainly  of  a  graphical  network  of  lesson-modules,  which 
depicts  what  lesson  modules  have  been  followed  by  the  students. 


Figure  7  Overview  of  session  progress 
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Object  oriented  approach  An  Important  part  of  user  interfaces  based 
upon  windows  is  direct  manipulation,  i.e.  the  fact  that  the  user  can 
manipulate  certain  objects  on  the  screen,  like  select,  move,  enlarge 
etc.  This  is  also  called  object  orientedness . 

The  instructor  module  user  interface  shows  the  object  orientedness  in 
different  ways.  For  example,  the  instructor  can  select  the  global 
overview  of  sessions  with  a  mouse.  By  selecting  a  session,  the 
contents  of  the  global  overview  is  specified  in  detail. 

The  lesson  modules  in  the  network  overviews  are  also  shown  as  objects 
which  can  be  manipulated.  By  selecting  a  module  (a  rectangle  in  a 
network,  see  figure  7)  a  pop-up  window  appears,  giving  more  detailed 
information  concerning  this  lesson  module. 

Because  of  the  many  positive  experiences  with  an  object  oriented 
approach  and  with  direct  manipulation,  these  items  will  be  used  more 
and  more  in  the  future  in  the  instructor  module. 

Evaluation  and  conclusions  Although  the  development  of  the 
instructor  module  and  the  UCM  is  still  in  progress,  a  number  of 
demonstrations  with  the  feasibility  model  were  given  and  a  first 
evaluation  with  expert  users  from  the  school  took  place.  The  first 
reactions  show  that  the  instructor  module  satisfies  the  requirements 
of  the  user,  the  instructor.  It  is  now  plausible  that  the  instructor 
module  is  sufficiently  universal,  i.e.  applicable  for  other  trainers. 

It  turned  out  that  the  instructor  module  is  a  good  step  in  the 
direction  of  an  instructor  communication  which  can  combine  the 
necessary  freedom  for  the  instructor  with  the  automatic  instructional 
process  of  the  UCM.  However,  some  guidance  of  the  instructor  is  still 
necessary  in  a  window  environment. 

In  the  future,  when  the  instructor  module  has  been  developed  into  a 
full  prototype,  more  attention  will  have  to  be  payed  to  the 
possibility  of  presenting  data  through  graphic  overviews,  together 
with  an  object-oriented  approach  of  the  user- interface . 
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COMPUTERIZING  A  JOB  SKILLS  INVENTORY 


Dr .  DeLayne  Hudspeth 

The  process  and  results  of  automating  an  Air  Force  Job  Skills 
Inventory  is  reviewed.  A  prototype  computerized  survey  was  used 
to  determine  that  computer  administration  is  feasible  and  seems 
to  yield  a  higher  number  of  job  task  selections  than  does 
traditional  administration.  Rationale  for  this  finding  are 
provided.  This  project  also  raised  a  number  of  research 
questions  both  of  the  survey  used  and  more  generally  in  terms  of 
systematically  using  computers  to  collect  test  and  questionnaire 
data.  Specific  questions  that  concern  the  moment  of  contact 
between  the  respondent  and  computer  administration  are  detailed. 
A  plan  for  using  a  Bulletin  Board  System  and  collaborative 
research  is  discussed. 

Dr.  DELAYNE  HUDSPETH  Associate  Professor  in  the  Area  of 
Instructional  Technology,  College  of  Education,  The  University  of 
Texas  at  Austin  (78712)  enjoys  teaching  a  variety  of  courses 
including  ITV  Studio  Production,  Telecommunications,  Analysis  of 
Research  in  Instructional  Technology,  Systematic  Design  of 
Courseware  and  Management  of  Training.  His  research  activities 
include  the  use  of  Bulletin  Board  Systems  to  support  teachers, 
Feedback  and  Feedforward,  Distance  Education  methods  and  the 
effect  of  Case  Studies  on  Higher  Level  Objectives.  He  recently 
received  a  Summer  Research  Fellowship  at  the  AF  Human  Resources 
Laboratory,  Division  of  Manpower  and  Personnel  at  Brooks  Air 
Force  Base,  which  provided  the  background  for  this  paper. 
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Computerizing  a  Job  Skills  Bank 

by 

Dr.  DeLayne  R.  Hudspeth 


Introduction 


The  Occupational  Measurement  Center  (OMC) ,  located  at  Randolph 
AFB,  Texas  is  responsible  for  the  preparation,  administration  and 
analysis  of  job  classification  surveys  within  the  USAF. 
Currently,  the  time  between  survey  mailing  and  initial  data 
processing  ranges  from  seven  to  nine  months.  The  OMC  judged  that 
current  methods  of  obtaining  and  processing  data  for  occupational 
analysis  studies  is  slow,  complicated,  and  expensive.  The  OMC 
subsequently  asked  that  the  AF  Human  Resources  Lab,  Manpower  and 
Personnel  Division,  (AFHRL/MOD)  investigate  a  quicker,  more 
efficient  system  of  administering  occupational  job  surveys.  Of 
particular  interest  was  the  possibility  of  automating  the  process 
through  personal  computers  and  the  use  of  the  DDN  for 
distribution. 

A  full  report  of  this  research  is  available  from  AFHRL/MOD 
(Brooks  AFB,  TX  78235)  and  is  only  briefly  reviewed  in  this 
paper.  The  purpose  of  this  presentation  is  to  describe  the 
developmental  stages  incurred  in  computerizing  a  data  bank  and  to 
describe  the  critical  issues  facing  technologists  as  we  move 
toward  automating  job  inventories,  tests,  questionnaires  and 
other  data  bases.  Traditional  forms  of  survey  and  test 
administration  offer  some  advantages  to  both  teacher  and  learner; 
and,  so  does  computer  administration.  Care  and  additional 
research  are  needed  to  optimize  computer  use. 

Objectives  of  the  Survey  Automation  Research  Project  Five 
objectives  were  defined  as  outcomes  for  my  summer  project:  1)  to 
create  a  computerized  version  of  the  Chapel  Management  Job  Survey 
(chosen  because  it  was  about  to  be  administered  via  traditional 
means) ;  2)  to  prepare  a  research  design  for  comparing  the  two 
forms  cf  administration;  3)  to  collect  data;  4)  to  analyze  the 
data  ana  describe  the  results;  and  5)  to  provide  recommendations 
for  R&D  based  on  our  insights  and  experience. 

Computer-based  Chapel  Management  Job  Inventory  A  computer-based 
job  inventory  was  developed  using  Microsoft's  QuickBasic  version 
4.5.  This  third  generation  language  is  internally  well 
documented  and  allowed  us  to  write,  test  and  place  software  in 
the  field  in  less  than  four  weeks.  Modular  development  and 
formative  evaluation  was  used  throughout  this  process. 

The  software  consists  of  two  independent  modules  that  are  chained 
together.  The  first  module  contains  the  Biographical  and 
Background  Sections,  has  13  subprocedures  and  2576  lines  of  code. 
The  second  module  is  the  Duty-Task  Section,  contains  18 
subprocedures,  1341  lines  of  code  and  serves  two  major  functions. 
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The  first  has  the  incumbent  reviewing  each  of  407  tasks  and 
identifying  the  ones  done  in  his  or  her  job.  The  second 
procedure  consists  of  the  job  incumbent  rating  relative  time 
spent,  with  a  nine  point  scale,  only  those  tasks  identified  in 
the  first  procedure.  In  both  procedures  the  incumbents  could 
back  up  and  review  or  change  answers  and  ratings. 

To  summarize,  information  generated  from  the  second  module 
included  the  identification  of  tasks  done  by  incumbents  in  their 
present  job  and  time  rating  data  for  each  task;  and  (for  research 
purposes)  amount  of  time  spent  in  using  the  module,  times  an 
incumbent  backed  up  and  changed  an  answer  and  how  many  error 
messages  were  written  to  the  screen.  All  data  were  written  out 
to  data  files  and  captured  on  floppy  disks. 

Research  Design  The  two  independent  variables  were  form  and 
sequence:  the  two  forms  were  paper/pencil  (P)  or  computer-based 
(C)  ;  and,  three  types  of  sequence,  1)  P  followed  by  P,  2)  P 
followed  by  C,  and  3)  C  followed  by  P.  (Note:  Although  the  fourth 
variation  of  C  followed  by  C  would  have  been  desirable  it  was  a 
condition  of  using  this  population  that  all  persons  must  take  a 
paper/pencil  survey.  We  judged  that  asking  airmen  to  take  the 
same  inventory  three  times  would  affect  reliability  of  the  data.) 

Test/re-test  procedures  were  used  for  comparing  the  two 
administrations  and  each  respondent  served  as  his  or  her  own 
control.  This  was  necessary  as  no  assumption  could  be  made  about 
the  central  tendency  and  dispersion  of  scores.  Each  respondent 
had  a  unique  pattern  of  responses  which  described  his  or  her  job. 

The  three  treatments  were: 

Time  1  fTl)  Time  2  (T2)  Original  N 


Treatment  #1 .  P  followed  by  P  (P-P)  40 

Treatment  #2 . P  "  '•  C  (P-C)  20 

Treatment  #3 .  C  ”  ”  P  (C-P)  21 


The  purpose  of  treatment  #1  (P-P)  was  to  provide  a  baseline 
against  which  the  other  treatments  could  be  compared  with  respect 
to  the  variability  of  test/re-test.  Although  a  perfect  match  was 
not  expected,  a  reasonably  high  match  was  anticipated,  as  jobs 
seldom  change  much  in  two  weeks.  (Time  between  all  treatments 
was  2-3  weeks.) 

The  P-C  and  C-P  treatments  provided  comparison  data  to  examine 
effects  due  to  form  of  administration.  For  example,  the  second 
survey  might  yield  a  higher  number  of  tasks  selected  than  the 
first,  since  the  first  administration  might  sensitize  incumbents 
to  the  nature  of  their  jobs.  However,  this  effect  could  be 
confounded  for  the  P-C  and  C-P  sequences  because  it  is  possible 
that  all  C  administrations  would  yield  a  higher  number  of 
responses  because  C  respondents  are  forced  to  look  at  each 
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separate  task  whereas  with  the  paper  version  they  might 
accidentally  scan  past  a  relevant  job  statement. 

Administration  and  Data  Collection  About  May  24,  (1990)  a 
printed  survey  (P)  which  included  the  usual  optical  scan  data 
function  was  sent  by  OMC  to  all  AF  bases  using  traditional  means 
and  methods  of  distribution.  The  first  incumbents  whose  surveys 
were  returned  to  OMC  were  immediately  sent  a  second  P 
administration  with  an  explanatory  letter.  By  July  5,  30  second 
returns  were  received  and  used  for  analysis. 

For  research,  printed  surveys  for  the  P-C  treatment  were  hand 
delivered  to  the  Survey  Control  Officers  (SCO)  at  Bergstrom, 
Kelly  and  Randolph  AF  bases.  The  computerized  version  was  then 
given  within  2-3  weeks  following  the  paper  administration.  For 
all  computerized  administrations,  local  Z-248  PC's  were  used  and 
scheduled  by  the  Chapel  POC.  Each  respondent  used  a  separate 
disk  for  taking  the  survey.  For  the  C-P  treatment  a  reverse 
process  was  used  starting  with  Brooks  and  Lackland  AF  bases.  All 
paper  versions  were  machine  scanned  by  OMC  to  create  a  data  file 
on  disk,  which  was  then  matched  and  merged  with  the  data  strings 
collected  via  the  PC's. 

Results  Because  the  full  data  are  reported  elsewhere  (see  above 
for  the  full  technical  report  due  from  AFHRL/MOD  early  1991;  and, 
in  the  proceedings  of  the  1990,  32nd  Annual  Conference  of  the 
Military  Testing  Association)  I  only  provide  summary  data  in 
Table  I  and  II.  Based  on  this  and  other  data  not  provided  here, 
I  will  briefly  summarize  my  personal  conclusions  of  this  project 
and  then  move  to  define  some  of  the  critical  issues  related  to 
computerized  survey  and  test  design  and  administration.  Because 
the  size  of  this  (convenience)  sample  is  small  and  central 
tendency  can  not  be  assumed,  generalization  from  the  data  is 
severely  limited. 

Perhaps  the  most  important  benefit  of  the  summer  research  project 
is  that  the  Air  Force  has  a  prototype  computerized  job  survey  to 
demonstrate  the  potential  of  automated  administration.  A 


Table  I 

Summary  of  Data 


N=30  N=17  N=17 


Treatment 

PI 

-  P2 

PI 

-  C2 

Cl 

-  P2 

Total  tasks  select. 

.  3,684 

3,878  . 

1,808 

1,950 

.2,418 

2,267 

Mean  each  Ss. 

123 

129  . 

106 

115 

.  142 

133 

Selected  both  adm. 

• 

81% 

82% 

75% 

Mean  change 

• 

+6.5%  . 

+8.4% 

-8.9% 

job  survey  was  administered  with  a  microcomputer  (and  could  have 
been  distributed  via  network) ,  the  data  were  captured 
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electronically  and  analysis  was  accomplished  in  a  few  hours 
without  the  need  to  handle  paper  questionnaires  and  optical  scan 
sheets.  Although  this  accomplishment  may  seem  trivial  to  those 
who  routinely  administer  questionnaires  and  tests  via  computer, 
in  complex  organizations  a  working  prototype  is  sometimes  useful. 

We  found  that  the  computer  version  consistently  captured  more  job 
tasks  than  did  paper  administration  (see  Table  I,  above).  This 
can  be  explained  when  considering  that  the  paper  version,  in  my 
opinion,  suffers  from  readability  factors,  and  it  is  easy  skip 
over  similar  looking  task  statements.  Whereas,  with  the  computer 
version  the  respondent  must  lock  at  every  task  statement,  and 
overtly  respond,  before  seeing  the  next  task  description. 

Another  research  issue  was  whether,  for  the  job  tasks  selected  by 
incumbents,  the  estimates  of  "Time  Spent  in  Present  Job”  would 
vary  as  a  function  of  type  of  administration.  (This  is  estimated 
with  a  nine  point  scale;  1  =  "Very  small  amount") .  Table  2 
shows,  where  the  same  job  task  was  identified  in  both 
administrations,  the  change  in  ratings  from  the  first 
administration  to  the  second.  For  example,  for  the  PI  -  P2 
administration  of  2,988  tasks  chosen  1,283  time  ratings  were  the 
same,  387  estimates  were  one  point  higher  for  more  time  spent; 
342  estimated  one  point  less  time  spent,  etc. 


Table  2 

Time  Spent  Estimates  for  Jobs  Identified  Both  Administrations 


N 

*  30 

N 

=  17 

N 

=  17 

PI 

-  P2 

PI 

-  C2 

Cl 

-  P2 

Change 

#  Of  Tasks 

Change 

#  of  Tasks 

Change 

#  of  Tasks 

-8 

0 

-8 

3 

-8 

9 

-7 

1 

-7 

2 

-7 

9 

-6 

2 

-6 

3 

-6 

16 

-5 

10 

-5 

11 

-5 

27 

-4 

74 

-4 

60 

-4 

74 

-3 

124 

-3 

41 

-3 

113 

-2 

239 

-2 

116 

-2 

229 

-1 

342 

-1 

154 

-1 

364 

0 

1,285  67% 

0 

560  64% 

0 

657  64% 

+  1 

387 

+1 

231 

+  1 

133 

+2 

281 

+2 

137 

+2 

89 

+3 

146 

+3 

84 

+3 

62 

+4 

76 

+4 

78 

+4 

19 

+5 

11 

+5 

6 

+5 

4 

+6 

8 

+6 

2 

+6 

6 

+7 

1 

+7 

0 

+7 

1 

+8 

T 

1 

=2,988 

+8 

0 

T=l,488 

+8 

0 

T=l,812 

If  one  assumes  that,  on  a  nine  point  scale,  it  is  reasonable  to 
vary  by  one  point  in  re-estimating  relative  time  spent  on  each 
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separate  job  task  statement,  we  find  that  for  the  three 
treatments  that  67%  (N=2,014),  64%  (N=945)  and  64%  (N=l,154)  were 
consistently  rated  from  the  first  to  the  second  administration. 
Although  the  distribution  of  these  re-estimates  generally 
resemble  a  normal  distribution,  further  analysis  of  why  some 
incumbents  varied  so  much  awaits  further  research,  probably  using 
qualitative  methods  such  as  "think  aloud"  protocols  or  other 
means  of  collecting  individual  reasons.  Also,  in  the  C1-P2 
treatment  there  seems  to  be  a  shift  toward  estimating  less  time 
with  paper.  A  replication  study  is  needed  to  see  if  this  shift  is 
spurious . 

I  would  also  note  in  passing  that  some  of  the  variability  we 
observed  in  estimating  the  relative  time  spent  on  each  job  task, 
may  have  been  a  function  of  survey  instructions.  Specifically, 
there  were  both  data  and  informal  feedback  to  suggest  that  the 
directions  for  estimating  "Time  Spent..."  could  have,  by  the 
incumbents,  more  than  one  meaning.  Subsequently,  we  recommended 
that  additional  research  address  this  issue. 

Summary  of  "Automation  of  Job  Surveys"  Recommendations  Briefly, 
we  recommended  that  the  USAF  begin  immediately  to  automate  the 
administration  of  job  inventories.  Full  automation  includes 
three  primary  issues.  First,  a  comprehensive  electronic  network 
needs  to  be  designed  that  will  allow  OMC  to  electronically 
distribute,  administer,  process  and  archive  occupational  surveys. 
"Personnel  Concept  III"  (PC-3)  currently  under  development  by 
AFMPC,  with  gateways  through  each  AF  CBPO,  should  be  investigated 
further.  Second,  because  the  computer  offers  display,  review  and 
reporting  capabilities  not  available  with  traditional  paper 
administration  we  strongly  recommend  that  planning  efforts  to  use 
this  capability  be  undertaken  as  soon  as  possible.  Specifically, 
a  policy  review  of  occupational  data  needs  might  surface  manpower 
reporting  needs  which  new  technology  could  address.  Third, 
research  is  needed  to  optimize  design  of  computerized  surveys 
which  might  include  branching  or  mapping,  differential  feedback 
based  on  individual  patterns  of  response,  procedures  for  review 
and  correction  of  responses  by  incumbents  and  other  survey  design 
functions  which  are  unique  to  the  computer. 

Computerizing  Tests  and  Questionnaires  During  preparation  of  the 
computer  based  job  inventory  described  above  it  became  apparent 
when  reviewing  the  literature  that  relatively  little  research  has 
been  done  to  help  optimize  the  design  of  computerized  tests  and 
questionnaires.  This  issue  is  especially  important  when 
considering  the  unique  potential  of  the  machine  to  process 
results  and  provide  differential  displays  based  on  user  input. 
Although  there  is  a  modest  base  of  psychometric  data,  and 
increasing  interest  in,  adaptive  testing,  this  is  but  one  of  many 
approaches  that  should  be  considered. 

My  interest  is  driven  by  two  concerns.  First,  I  am  concerned 
that  some  persons  will  presume  that  what  works  well  with  paper 
and  pencil  will  work  equally  well  with  the  computer.  I  suggest 
that  this  transfer  process  poses  very  real  issues  of  reliability 
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and  validity.  Second,  I  am  concerned  that  the  potential  of 
automated  data  capturing  will  be  lost  if  we  do  not  carefully 
think  through  the  design  and  administration  issues  of  automated 
testing  and  carefully  document  the  cost/benefits.  The  habits 
that  we  form  early  are  hard  to  break  later,  even  apart  from  the 
technical  issues  of  data  file  standards,  etc. 

One  way  to  approach  the  problem  is  to  consider  a  given  test  or 
questionnaire  as  the  hub  of  a  system.  This  system  can  be  defined 
simplistically  as  the  institutional  need  (represented  by  the 
teacher  or  designer  of  the  test  or  questionnaire) ,  the  medium 
used  such  as  a  computer,  the  testee  and  his  or  her  interface 
with  the  medium,  feedback  to  both  the  testee  and  tester,  and 
effect  of  the  data  on  the  system. 

For  the  purpose  of  this  presentation  let  me  focus  just  on  the 
person  taking  the  test,  the  testee,  and  those  moments  of  contact 
wherein  a  CRT  display  and  keyboard  are  used  to  capture  the 
knowledge,  skill  and  attitudes  of  a  person  whose  promotion  or  job 
may  be  on  the  line.  Research  questions  include: 

-  What  are  the  minimum  keyboard  skills  needed?  Do  these 
skills  interact  with  the  type  of  test  given  (such  as 
multiple  choice,  essay,  constructed  response,  matching)? 

-  Does  the  respondents  attitude  toward  a  computer  affect 
the  reliability  of  responses  (Hawthorn  effect?  Computer 
phobia?) 

-  Is  readability  the  same  for  a  CRT  as  for  print  material; 
for  monochrome  and  color;  equal  for  all  standards  such 
as  EGA,  VGA? 

-  What  are  optimum  construction  guidelines  for  number  of 
items,  length  of  sentence,  placement  of  distractors? 

-  What  are  the  effects  of  different  response  modes  (cursor 
control;  A,B,C  vs.  1,2,3;  forced  choice  vs.  open  response). 

-  If  scales  or  graphs  are  used  on  the  CRT  is  some  level  of 
graphic  literacy  required?  Can  print-based  measures  be 
applied  to  CRT  display? 

-  If  pictures  are  used  on  the  CRT  as  a  source  of  test 
material  how  does  limited  resolution  and  color  shifts 
affect  the  ability  of  the  testee  to  discriminate;  how 
does  this  interact  with  the  purpose  of  the  question? 

-  Testees  sometimes  scan  an  entire  test,  before  responding, 
for  an  overview  and  for  pacing.  Does  lack  of  this 
ability  affect  some  testees  more  than  others?  Affect 
distribution  of  results? 
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-  On  most  tests  and  questionnaires  the  testee  can  review 
and  change  an  answer.  Is  reverse  scrolling  and  answer 
changing  important?  For  whom?  Why? 

-  A  computer  can  analyze  data  and  create  unique  displays 
for  summary  review  and  "map”  appropriate  items  to  add 
or  delete.  What  are  guidelines  for  using  this  capacity? 
How  is  scoring  accomplished  with  multiple  pathways? 

How  is  reliability  of  such  tests  determined? 

The  list  could  go  on.  The  point  is  that  innumerable  questions 
can  and  should  be  raised,  some  more  important  than  others, 
concerning  our  ability  to  use  computers  for  data  collection. 
Further,  when  we  consider  the  rest  of  the  system  and  supra-system 
needs,  we  move  into:  standards  of  design;  data  transmission 
requirements;  operating  system  needs  to  suit  various  equipment 
bases,  archival  standards  for  both  the  instruments  and  the  data, 
up-grade  requirements  for  better  and  faster  equipment  (such  as 
high  definition  TV)  and,  a  host  of  other  issues. 

I  have  chosen  to  focus  on  that  moment  of  contact  between  the 
respondent  and  the  machine  and  in  the  future  will  be  proposing  a 
program  of  research  to  address  the  specific  issues  of  man-machine 
interface.  I  anticipate  continued  collaboration  with  the  Air 
Force  Human  Resources  Lab  at  Brooks  Air  Force  Base  and  propose 
that  we  establish  a  special  interest  group  to  work  within  and 
between  institutions.  By  December  1,  1990  I  hope  to  have  a 

Bulletin  Board  System  that  can  be  used  to  share  and  develop 
ideas,  coordinate  research  efforts  and  provide  a  data  base. 

The  next  step  is  to  review  and  evaluate  existing  research  to 
determine  what  works.  From  this  data  base  we  can  determine  what 
is  needed  and  create  a  long  range  program  of  research  efforts  so 
that  each  one  of  us  might  address,  with  minimum  redundancy,  some 
aspect  of  the  overall  need.  The  problem  is  a  bit  like  eating  an 
elephant,  the  first  task  is  to  carve  it  into  manageable  chunks! 
Finally,  I  am  proposing  the  development  of  a  research  tool  or 
"shell"  that  will  allow  researchers  to  select  from  a  menu  various 
dependent  and  independent  variables  and  have  these  automatically 
formatted  for  computer  administration.  I  hope  to  report  on  this 
effort  at  the  next  TITE  conference. 

Summary  of  Issues  Facing  Computerization  Relatively  little 
research  is  available  which  can  be  used  to  guide  the  design  of 
computerized  tests  and  questionnaires.  Elements  of  this  issue 
involve  format  issues  (such  as  the  optimum  number  of  items 
displayed  at  one  time,  the  effect  of  scrolling,  optimum  placement 
and  format  of  instructions) ;  the  effect  of  prior  computer 
experience  (keyboard  skills,  fear  of  using  a  computer) ;  scaling 
methods  (optimum  scaling  techniques  which  use  the  medium  of  CRT 
and  keyboard) ;  and,  the  costs  and  benefits  of  using  the  unique 
potential  of  the  computer  (such  as  mapping  or  summary  reviews 
based  on  a  pattern  of  responses) . 
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Proposed  A  special  interest  group  is  proposed  that  would 
communicate  with  a  Bulletin  Board  System  (further  information  can 
be  obtained  via  BitNet;  DeLayneH§UTXVM) .  Current  efforts  include 
a  review  of  research  to  determine  what  works;  collaboration  to 
address  gaps  in  this  data  base;  and,  the  design  of  a  research 
productivity  tool  or  "shell”  that  will  allow  researchers  to 
automatically  format  computer  based  tests  and  questionnaires  for 
electronic  delivery  and  rapid  data  analysis. 

Note:  This  paper  is  based,  in  part,  on  research  conducted  with  a 
1990  USAF-UES  Summer  Faculty  Research  Program  grant  at  the  USAF 
Human  Resources  Laboratory  in  San  Antonio,  Texas.  I  wish  to 
thank  the  Air  Force  Systems  Command  and  the  AFHRL,  Manpower  and 
Personnel  Division  for  this  support.  Also,  contributing  to  this 
effort  was  Mr.  Paul  Fayfich,  graduate  student  in  Instructional 
Technology,  at  the  University  of  Texas  at  Austin. 
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EVALUATING  TRAINEE  PERFORMANCE 
ON  SIMULATION  TRAINING  SYSTEMS 

John  E.  Biegel,  Murat  Draman,  Catharina  Eeltink 

Intelligent  Simulation  Training  Systems  require  an  Evaluator  module  that  will 
automatically  and  intelligently  assess  trainee  performance.  In  some  domains  a 
trainee’s  response  can  be  categorized  dichotomously,  e.g.  right  or  wrong.  In  most 
domains,  particularly  in  those  domains  where  training  is  accomplished  using 
simulations,  there  may  be  several  correct  trainee  actions  for  each  and  every  scenario. 
Some  responses  may  reflect  more  advanced  levels  of  skill  acquisition  than  others.  In 
this  situation  the  Evaluator  must  score  the  trainee’s  response  relative  to  a  set  of 
possible  correct  actions  which  an  expert  would  find  acceptable. 

The  University  of  Central  Florida,  Embry-Riddle  Aeronautical  University  and 
the  General  Electric  Company  at  Daytona  Beach  are  working  together  to  build  an 
Intelligent  Simulation  Training  System  (ISTS).  Currently,  it  is  in  a  prototype  state. 
The  demonstration  domain  is  Air  Traffic  Controller  (ATC)  training.  One  of  the 
significant  tasks  has  been  to  build  the  Evaluator  module.  The  Evaluator  takes  an 
input  from  the  trainee,  compares  it  to  a  set  of  actions  that  an  expert  ATC  might  take, 
and  makes  an  evaluation. 

Dr.  JOHN  E.  BIEGEL  A  Professor  of  Engineering,  Dr.  Biegel  is  Director  of  the 
Intelligent  Simulation  Laboratoiy,  and  Principal  Investigator  for  the  Intelligent 
Simulation  Training  Systems  project  in  the  Department  of  Industrial  Engineering  and 
Management  Systems  at  the  University  of  Central  Florida.  Dr.  Biegel  holds  degrees 
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Introduction 

Intelligent  Training  Systems  (ITS) 
are  the  way  of  the  future  for  training 
in  motor-skill  tasks.  Many  of  these 
systems  will  be  simulation-based.  In  a 
broad  and  generic  sense,  we  can  view 
the  objective  of  simulation-based 
training  systems  as  training  for  the 
proper  manipulation  of  objects  in  time 
and  space.  This  is  true  foi  training  a 
missile  operator,  an  air  traffic 
controller  (ATC),  a  sonar  operator,  a 
tank  driver,  a  truck  driver,  etc.  Other 
domains  where  the  same  or  similar 
principles  apply  involve  diagnostics  or 
operator  training  for  a  power  plant  In 
some  of  these  cases  the  simulation 
training  process  involves  only  the 
identification  of  irregular  conditions  or 
abnormal  conditions  and  the  proper 
corrective  actions.  In  a  sense,  ATC 
training  might  be  considered  one  of 
the  most  difficult  because  the  trainee 
and  the  simulator  aUemately  drive  the 
process.  (When  the  trainee  enters  an 
acceptable  input,  the  simulator  uses 
the  new  input  as  a  modification  to  the 
direction  in  which  to  proceed  until  the 
next  input).  The  simulator  may  also 
be  internally  modified  by  the  tutorial 
component  to  alter  a  simulation 
scenario.  This  is  done  dynamically,  as 
a  system  reaction  to  the  trainee’s 
current  performance.  The  Evaluator’s 
task  is  to  determine  relative 
correctness  and  quality  of  the  trainee’s 
input  The  process  of  evaluation 
expands  with  each  added  rule  and  each 
added  feature  of  the  system.  (Note: 
At  present  no  cognitive  evaluation  of 
learning  styles  or  reasoning  processes 
is  carried  out  The  Evaluator  only 
assigns  a  quantitative  score  to  the 
trainee’s  response  based  on  what  it 
accomplishes.) 


The  Intelligent  Simulation  Training 
System  fISTS’l 

The  Intelligent  Simulation 
Training  System  (ISTS)  is  a  generic 
system  with  eight  generic  components 
and  two  domain  dependent  com¬ 
ponents.  The  generic  components  are 
the  Control,  Inference  Engine,  Transla¬ 
tor,  Input  Filter,  Interface,  Tutor,  Stu¬ 
dent  Model  and  Evaluator.  The  Do¬ 
main  Expert  (DE)  and  the  Domain 
Expert  Instructor  (DEI)  contain  do¬ 
main-specific  knowledge  bases.  The 
DE  knowledge  base  is  partitioned  into 
two  components:  a  component  that 
detects  events  that  require  trainee 
intervention  (the  Dc  'uain  Expert  Prob 
lem  Finder  or  the  Intelligent  Pre-Pro¬ 
cessor)  and  a  component  that  gener¬ 
ates  suggested  actions  (the  Domain 
Expert  Problem  Solver).  The  DEI 
incorporates  all  domain-specific  infor¬ 
mation  for  creating/modifying  a  scenar¬ 
io  and  for  evaluating  the  actions  of  the 
trainee.  The  generic  components- 
/modules  that  need  domain  knowledge 
(i.e.  the  Tutor  and  the  Evaluator) 
obtain  it  from  the  DEI  and  the  DR 
The  simulator,  of  course,  must  be 
progranuned  to  display  the  proper 
objects  and  their  appropriate  actions, 
identification,  parameter  values,  etc. 
(A  generic  simulator  could  be  prog¬ 
ranuned  in  an  object-oriented  language 
so  that  the  required  parameter  values, 
etc.  are  instantiated  at  initialization.) 

The  purpose  of  ISTS  is  to  take  over 
the  active  training  duties  of  an 
instructor.  The  instructor’s  role  will  be 
that  of  a  learning  manager  who  estab¬ 
lishes  the  content  (types  of  events  that 
are  traced,  difficulty  level  of  the  sce¬ 
nario,  etc.)  and  evaluation  principles 
(topic  weights,  importance  of  events 
and  their  status,  etc.)  of  a  planned 
training  session.  Once  a  trainee  starts 
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a  session,  the  Tutor  component  of  the 
ISTS  takes  over  to  integrate  the 
instructor’s  criteria. 

The  Training  Components 

The  modules  within  the  ISTS 
which  directly  assess  trainee  perfor¬ 
mance  are  the  Tutor,  the  Student 
Model  and  the  Evaluator.  These  three 
components  act  in  a  cooperative  man¬ 
ner  to  adequately,  efficiently  and  effec¬ 
tively  train  the  trainee.  These  modules 
may  utilize  knowledge  residing  in  the 
DE  and  the  DEI,  but  the  DE  and  The 
DEI  do  not  react  to  trainee  inputs 
directly.  The  role  of  the  Tutor  is  to 
determine  the  appropriate  training  for 
the  trainee  at  all  points  in  the  training 
process.  To  do  this,  the  Tutor  obtains 
information  on  the  trainee’s  status 
from  the  Student  Model,  and  evaluates 
that  information  in  light  of  the  trai¬ 
nee’s  current  oerformance,  past  perfor¬ 
mance,  the  difficulty  of  the  scenario, 
the  objective  of  the  current  lesson,  etc. 
The  Tutor  then  decides  to  decrease, 
increase,  or  maintain  the  intensity  and 
difficulty  of  the  training  based  upon 
the  above  evaluation  process.  At  times 
it  provides  immediate  feedback  to  the 
trainee.  It  can  aiso  provide  "help"  and 
"advice". 

The  role  of  the  Student  Model  is 
a  benign  one  of  record  keeping.  The 
past  performance  record  is  loaded  at 
initialization.  The  Student  Model 
updates  the  trainee  record  as  the  train¬ 
ing  process  proceeds  and  saves  it  at 
the  end  of  a  session. 

The  role  of  the  Evaluator  is  to 
first  score  the  trainee  input  in  terms  of 
its  domain  appropriateness  (see  next 
paragraph).  If  the  inp’:+  is  accepted, 
the  Evaluator  t^ca  -evaluates  the 
input’s  effectiveness  relative  to  what  an 


expert  would  do  under  the  same 
conditions. 

Determining  the  Acceptability  of  an 
Input 

There  are  several  tests  which  a 
tiainee’s  input  must  pass  prior  to  its 
being  sent  to  the  simulator  for 
execution.  An  input  which  does  not 
pass  these  preliminary  tests  is  rejected, 
scored  as  a  failure,  and  no  further 
processing  is  necessary.  (Note:  The 
following  tests  are  specific  to  the  ATC 
domain,  but  may  apply  more  broadly.) 

The  first  test  on  the  trainee  input 
is  for  correct  spelling.  Incorrect 
spelling  must  be  evaluated  while  the 
trainee  is  learning  the  keyboard.  At 
present  inputs  are  entered  by 
keyboard.  This  adds  a  component  to 
the  evaluation  process  which  would  not 
be  there  if  we  were  using  voice  input, 
a  future  enhancement.  After  this 
initial  learning  phase,  incorrect  spelling 
must  be  rejected  without  affecting  the 
trainee’s  record.  Such  a  situation  will 
not  be  encountered  with  voice  or  menu 
driven  interfaces. 

If  the  spelling  is  correct,  the  input 
is  tested  is  for  proper  syntax.  Syntax 
errors  reflect  deviations  from  the 
precise  command/input  format 
required  by  a  particular  domain.  The 
significance  of  syntax  errors  could  vary 
depending  upon  the  lesson  objective. 

The  third  test  is  for  the 
appropriateness  of  the  input  to  the 
physical  environment:  Does  it  conform 
to  speed  restrictions,  altitude 
restrictions,  forbidden  air  space,  (etc.)? 

The  fourth  test  is  for 
appropriateness  with  respect  to  the 
object;  i.e.  can  the  airplane  achieve  the 
given  speed,  altitude,  etc.?  This  test 
applies  only  in  a  general  sense  since  a 
controller  cannot  be  expected  to  know 
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all  of  the  characteristics  of  all  models 
and  versions  of  all  aircraft  under  all 
load  and  weather  conditions. 

Additionally,  this  test  also  verifies 
logic  errors:  the  trainee  may  ask  a 
plane  to  descend  to  12000  feet,  while 
the  plane  is  already  cruising  at  10000 
feet. 

The  fifth  test  concerns  whether 
the  pilot  accepts  the  input.  The  pilot 
can  reject  an  instruction  (for  example, 
being  told  to  fly  directly  into  severe 
weather),  although  normally  instruc¬ 
tions  are  accepted. 

An  input  that  fails  one  of  the 
above  tests  is  rejected  with  an 
explanatory  message.  The  Evaluator 
then  assigns  negative  points  under 
phraseology-error,  environment-error, 
logic-error,  argument-error,  or  object- 
error  categories. 

Tracing  the  Effects  of  a  Trainee  Input 

Once  an  input  has  passed  the 
above  tests,  it  is  translated  into  a 
format  that  the  simulator  understands. 
The  system  constantly  keeps  track  of  a 
"snapshot"  of  impending  events 
associated  with  each  object  (plane)  in 
the  simulator.  A  new  "snapshot"  of 
impending  events  for  the  modified 
object  is  taken  after  the  trainee  input 
is  implemented.  This  new  list  is 
compared  to  the  previous  list  of  events 
which  involve  the  modified  object. 
The  difference  shows  the  effects  of  the 
trainee  input  on  the  simulator.  The 
trainee  input  is  then  evaluated  based 
on  the  impending  events  that  are 
eliminated,  unaffected  and  created. 
The  type,  quantity,  and  status  of  these 
events  are  taken  into  consideration.  If 
an  input  affects  none  of  the  events,  it 
is  an  ineffective  input.  Typical  events 
in  air  traffic  control  are  potential 
safety/separation  violations  between 


airplanes,  emergencies,  handoff 
procedures,  etc.  Each  event  represents 
a  situation  that  the  trainee  must 
eliminate,  or  a  procedure  that  must  be 
executed.  There  is  a  one-to-one 
correspondence  between  the  current 
set  of  events  in  a  simulation  scenario 
and  the  responses/actions  required  of 
the  trainee.  Each  event  has 
appropriate  time  tags  that  are  used  in 
determining  the  status  of  the  event. 
One  time  tag  represents  the  latest  time 
by  which  they  must  be  eliminated  in 
order  to  avoid  trainee  intervention 
delays  and  critical/fatal  events. 
Another  time  tag  marks  the  detection 
time  for  the  event.  This  tag  is  used  for 
determining  how  long  the  event  is  left 
unattended,  or  how  long  it  took  the 
trainee  to  eliminate  it.  An  instructor 
(in  the  classroom)  specifies  the  time 
interval  to  be  used  for  determining 
when  the  event  reaches  a  "critical" 
status.  An  event  is  given  a  "late"  status 
if  it  is  still  present  after  its  latest- 
elimination  time. 

A  typical  trainee  input  may:  (1) 
eliminate  one  or  more  events  ("simple 
elimination"  input),  (2)  create  one  or 
more  events  ("simple  creation"  input), 
(3)  eliminate  events  while  creating 
other  events  ("side-effect"  input),  or  (4) 
cause  no  changes  in  the  snapshot 
("inefficient"  input).  The  input  is 
classified  into  one  of  these  four 
categories. 

Evaluating  a  Trainee  Input 

For  each  type  of  event,  the 
instructor  specifies  a  starting  score,  and 
a  series  of  correction  factors.  These 
are  used  for  determining  the  number 
of  points  to  be  added  to  or  deducted 
from  the  current  score  for  a  type  of 
event,  each  time  such  an  event  is 
found  to  be  created  or  eliminated. 
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With  a  negative  point  value,  the 
instructor  may  specify  that  the  creation 
or  elimination  of  a  type  of  event  is 
undesirable.  As  an  example,  creating 
a  potential  separation  violation 
between  two  airplanes  is  considered 
undesirable.  The  trainee  may  be  given 
positive  points  for  eliminating  such  an 
event  before  it  becomes  critical. 

The  correction  factors  are  used 
for  distinguishing  the  scoring  for 
new/eliminated  events  based  on  their 
status: 

non-critical  events  created  by  a 
simple-creation  input, 
immediately  critical  events 
created  by  a  simple -creation 
input, 

immediately  late  events  created 
by  a  simple-creation  input, 
non-critical  events  created  by  a 
side-effect  input, 
immediatel'  critical  events 
created  by  a  side-effect  input, 

.  immediately  late  events  created 
by  a  side-effect  input, 
non-critical  tutor-created  event 
eliminated  without  side-effects, 
critical  tutor-created  event 
eliminated  without  side-effects, 
non-critical  tutor-created  event 
eliminated  with  side-effects, 
critical  tutor-created  event 
eliminated  with  side-effects, 
non-critical  trainee-created  event 
eliminated  without  side-effects, 
critical  trainee-created  event 
eliminated  without  side-effects, 
non-critical  trainee-created  event 
eliminated  with  side-effects, 
critical  trainee-created  event 
eliminated  with  side-effects, 
neglected  (late)  event  initially 
created  by  a  simple-creation- 
input. 


neglected  (late)  event  initially 
created  by  a  side -effect  input,  and 
neglected  (late)  event  initially 
created  by  the  Tutor. 

The  trainee  actions  are  also 
evaluated  by  comparison  to  expert 
suggestions.  Tne  trainee  action  with 
undesirable  side  effects  will  not  have 
its  match  in  the  set  of  expert 
suggestions.  A  trainee  action  without 
side-effects  will  match  one  of  the 
expert  suggestions  that  are  ordered  by 
preference  and  efficiency.  This 
ordering  is  accounted  for  in  the 
evaluation,  as  the  system  determines  if 
the  trainee  utilized  the  most  preferred 
solution  or  a  less  desirable  alternative. 
An  apparently  correct  trainee  action 
without  a  match  in  the  expert 
suggestion  set  may  be  considered 
imperfect  and  evaluated  accordingly. 

Inefficient  inputs  are  classified 
into  four  categories: 

1.  inefficient  inputs:  there  are  no 
events  (therefore  no  required 
interventions)  associated  with  the 
modified  object.  The  trainee  is 
focusing  on  an  object  that  does 
not  require  any  attention. 

2.  No-match  inputs:  there  is  no 
similarity  (no  key-word  match) 
between  the  trainee  input  and  a 
suggested  expert  action.  The 
trainee  tried  to  use  an 
unrecommended  instruction  to 
eliminate  an  existing  event. 
Example:  the  trainee  climbed  the 
plane  to  a  new  altitude  without 
successfully  eliminating  any  of  the 
pending  problems,  while  the 
expert  suggests  turning  the  plane 
to  a  new  heading, 

3.  Partial-match  inputs:  there  is  a 
key-word  match,  but  no  argument 
match.  The  trainee  used  a 
proper  action,  but  with 
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specifications  that  are  not  in  the 
suggested  range.  Example:  The 
expert  suggests  climbing  the  plane 
to  an  altitude  between  18000  and 
20000  to  eliminate  a  pending 
violation,  the  trainee  asked  the 
plane  to  climb  to  17000. 

4.  Perfect-match  inputs:  there  are 
key-word  and  argument  matches 
between  the  suggested  expert 
action  and  the  trainee  input.  The 
input  should  have  eliminated  the 
pending  event  There  is  a 
problem  with  the  Domain  Expert. 
The  event  is  not  detected 
properly,  or  the  expert 
suggestions  are  not  correct. 

The  instructor  may  specify  points  to  be 
deducted  for  the  first  three  categories 
of  inefficient  inputs.  No  evaluation  is 
done  for  the  forth  category. 

Similarly,  if  an  input  creates  side- 
effect  events,  the  comparison  with  the 
expert  actions  list  should  not  produce 
a  perfect-match.  If  the  trainee 
executed  an  action  suggested  by  the 
expert,  no  side  effects  should  have 
been  caused.  The  side-effect  events 
are  not  detected  properly,  or  the 
expert  suggestions  are  not  correct.  No 
evaluation  may  be  done  in  this  case. 

Separate  scores  are  kept  for  (a) 
each  type  of  event  traced  by  the 
Simulator,  (b)  the  input  errors  done  by 
the  trainee,  and  (c)  the  ineffective 
inputs.  The  final  score  is  a  weighted 
average  of  the  scores  in  these  different 
areas  with  the  weights  being 
determined  by  the  instructor. 

Conclusion 

The  Evaluation  strategy  explained 
above  is  completely  generic.  It  may  be 
applied  to  any  domain  which  involves 
temporal-spatial  reasoning.  Events 
may  be  created  or  eliminated. 


Creation  or  elimination  of  events  may 
be  desirable  (correct)  or  undesirable 
(incorrect)  outcomes  of  the  trainee’s 
manipulation  of  the  objects  in  the 
simulation.  The  trainee’s  interventions 
are  compared  to  a  Domain  Expert’s 
set  of  possible  correct  interventions 
and  scored  according  to  how  closely 
the  trainee’s  input  matches  the 
Expert’s  solutions.  The  scoring 
remains  generic  because  the  point 
values  assigned  to  trainee  actions  are 
initially  provided  by  the  Domain 
Expert  Instructor.  The  Evaluator  does 
not  contain  any  domain-dependent 
information.  It  only  has  the  general 
structures  needed  for  tracing  and 
assessing  the  possible  results  of 
procedures  which  could  conceivably  be 
carried  out  on  objects  located  in  space 
and  time.  Having  defined  these 
general  procedures,  the  specific  values 
associated  with  them  are  determined 
by  the  instructor,  integrated  in  the 
Domain  Expert  Instructor,  and  sent  to 
the  Evaluator  at  run-time.  This 
evaluation  method  can  therefore  be 
utilized  in  any  appropriate  domain  that 
provides  the  required  information. 

Current  work  in  the  Evaluator  is 
directed  towards  the  possibility  of  the 
inclusion  of  student  performance  data 
in  the  evaluation  process.  This  would 
personalize  the  evaluation  in  domains 
where  each  student  needs  to  be 
evaluated  differently. 

Another  area  of  research  is  the 
implementation  of  procedures  which 
will  enable  the  Evaluator  to  deduce 
what  reasoning  strategy  is  being  used 
by  the  student  in  his  problem  solving 
approach  (cognitive  evaluation).  This 
type  of  cognitive  evaluation  would  go 
beyond  quantitative  scoring  of 
responses  and  arrive  at  a  qualitative 
assessment  of  the  student’s  cognitive 
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style,  information  processing  abilities, 
types  of  data  being  attended  to,  and 
hypotheses  being  generated  during  the 
course  of  solving  the  problem 
situations  presented  in  the  simulation. 
Such  an  assessment  would  provide  a 
measure  of  the  student’s  level  of 
expertise  in  the  domain,  more  in  terms 
of  answering  the  question  "Does  the 
student  reason  like  an  expert  in  this 
field?",  or  "Does  the  student  approach 
the  problem  in  a  manner  similar  to 
how  an  expert  would  approach  the 
problem?",  in  addition  to  addressing 
the  question  of  how  closely  the 
student’s  responses  match  a  set  of 
Expert  solutions. 
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LEARNER  CHARACTERISTICS  INVOLVED  IN  DISTANCE  LEARNING 


Ann  T.  Cernicek  and  Heidi  A.  Hahn 


Distance  learning  represents  a  strategy 
for  leveraging  resources  to  solve  educa¬ 
tional  and  training  needs.  Although 
many  distance  learning  programs  have 
been  developed,  lessons  learned  regard¬ 
ing  differences  between  distance  learning 
and  traditional  education  with  respect  to 
learner  characteristics  have  not  been  well 
documented.  Therefore,  we  conducted  a 
survey  of  20  distance  learning  profes¬ 
sionals.  The  questionnaire  was  dis¬ 
tributed  to  experts  attending  the  second 
Distance  Learning  Conference  sponsored 
by  Los  Alamos  National  Laboratory. 

TTiis  survey  not  only  acquired  demo¬ 
graphic  information  from  each  of  the  re¬ 
spondents  but  also  identified  important 
distance  learning  student  characteristics. 
Significant  distance  learner  characteris¬ 
tics,  which  were  revealed  statistically  and 
which  influence  the  effectiveness  of  dis¬ 
tance  learning,  include  the  following; 
reading  level,  student  autonomy,  and  self- 
motivation.  Distance  learning  cannot  be¬ 
come  a  more  useful  and  effective  method 
of  instruction  without  identifying  and 
recognizing  learner  characteristics.  It  will 
be  important  to  consider  these  charac¬ 
teristics  when  designing  all  distance 
learning  courses.  This  paper  will  report 
specific  survey  findings  and  their  implica¬ 
tions  for  developing  distance  learning 
courses. 


ANN  T.  CERNICEK  is  a  staff  research 
assistant  in  the  Lab-wide  Strategic  Sys¬ 
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cultural  variables  that  influence  the 
effectiveness  of  distance  learning.  She 
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HEIDI  A.  HAHN  is  the  group  leader  in 
the  Systems  Performance  and  Analysis 
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gineering  and  an  M.S.  in  psychology,  both 
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is  interested  in  human  performance  and 
cognitive  strategies  in  complex  environ¬ 
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oping  and  conducting  distance  training 
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Learner  Characteristics  Involved  in  Distance  Learning 
Ann  T.  Cemicek  and  Heidi  A.  Hahn 


Introduction  Ever  since  the  one  room 
school  house,  it  has  been  evident  that 
numerous  student  variables  are  involved 
in  education.  Because  the  traditional 
classroom  has  been  around  so  long,  we 
have  been  able  to  identify  and  ac¬ 
commodate  several  of  these  variables. 
Recently,  because  of  a  technological  rev¬ 
olution,  a  new  type  of  education  has 
evolved.  It  is  known  as  distance  educa¬ 
tion,  distance  learning,  and  distributive 
learning.  Putting  semantics  aside,  they  all 
mean  the  same  thing. 

Distance  learning  is  structured  learning 
without  the  physical  presence  of  an  in¬ 
structor  (Los  Alamos  National  Labora¬ 
tory  Distance  Learning  Conference, 
10/89).  Garrison  and  Shale’s  definition 
from  the  American  Journal  of  Distance 
Education  says  that  distance  education 
implies  that  the  mtyority  of  educational 
communication  between  (among)  teacher 
and  student(s)  occurs  noncontiguously 
(Sewart,  Keegan,  and  Hoimberg,  1988). 
Distance  learning  is  independent  of  time 
and  place.  It  includes  print  media,  video, 
satellite  broadcast,  teleconferencing,  au¬ 
dio  graphics,  as  well  as  many  other  medi¬ 
ums.  Student  variables,  which  need  to  be 
identified  and  recognized  to  make  it  an 
effective  means  of  education,  are  con¬ 
comitant  with  this  new  technology. 

The  need  to  explore  alternative  means  of 
providing  education  cannot  be  underes¬ 
timated  as  the  U.S.  is  currently  undergo¬ 
ing  an  education  crisis.  This  crisis  en¬ 
compasses  many  factors,  which  include 
declining  standardized  test  performance, 
the  increasing  number  of  functionally  il¬ 
literate  people  in  the  nation,  and  a  per¬ 
ceived  apathy  toward  learning  in  general. 
The  inefficiencies  in  the  current  educa¬ 
tion  system  are  symptoms  of  the  ongoing 
education  crisis.  A  single  approach  will 
not  be  sufficient  to  solve  the  education 
crisis;  a  series  of  solutions  will  be  neces¬ 
sary. 


Thus,  the  objective  of  this  study  was  to 
explore  the  effectiveness  of  distance 
learning  as  an  alternative  to  traditional 
classroom  instruction.  Specifically,  the 
study  aimed  to  identify  those  student 
variables  needed  for  distance  learning  to 
be  successful.  What  are  the  learner  vari¬ 
ables  that  affect  the  effectiveness  of  dis¬ 
tance  learning?  What  are  the  percep¬ 
tions  of  experts  in  the  field  of  distance 
learning  involving  learner  variables? 
Student  variables  must  be  recognized, 
identified,  and  accommodated  for  dis¬ 
tance  learning  to  be  successful. 

A  review  of  the  distance  learning  litera¬ 
ture  showed  that  a  great  deal  of  research 
has  been  conducted.  However,  this  re¬ 
search  is  very  fragmented;  studies  differ 
with  respect  to  scientific  rigor  and  the 
adequacy  of  reporting  of  information  and 
are  sometimes  inconsistent  with  respect 
to  the  conclusions  drawn.  Thus,  although 
it  is  possible  to  compile  a  comprehensive 
list  of  variables  that  may  impact  the  effec¬ 
tiveness  of  distance  learning,  it  is  very  dif¬ 
ficult  to  assess  their  relative  importance. 

Therefore,  the  goal  of  this  report  is  to  as¬ 
sess  the  relative  importance  of  learner 
variables  on  the  effectiveness  of  distance 
learning.  Experts  in  distance  learning 
were  asked  to  respond  to  a  survey  in 
which  they  rated  the  importance  of  a 
large  number  of  variables  to  the  ef¬ 
fectiveness  of  distance  learning  (versus 
that  of  a  traditional  classroom).  Vari¬ 
ables  included  in  the  survey  were  selected 
from  the  literature  and  are  described 
below; 

■  reading  level/education  level  -  Find¬ 
ings  show  a  positive  relationship  be¬ 
tween  education  level  of  the  student 
and  completion  of  a  distance  learn¬ 
ing  course  (Woodley  and  Parlett, 
1983);  reading  level  may  impact  the 
student’s  ability  to  process  course 
materials. 


455 


■  gender  -  Some  research  claims  there 
are  sex  differences  with  males  having  a 
higher  dropout  rate  than  females 
(Duby  &  Giltrow,  1978),  but  the  rea¬ 
sons  for  this  result  are  not  known 
(Schurman  &  Blackman,  1989). 

■  age  -  Findings  show  that  students  in 
their  late  twenties  and  older  are  less 
likely  to  drop  out  of  a  course  than 
younger  students  are  (Wells,  1990). 

■  socioeconomic  status  -  Low  socioeco¬ 
nomic  status  individuals  may  have  dif¬ 
ficulty  with  independent  learning;  oc¬ 
cupation  and  educational  level  may  in¬ 
teract  with  socioeconomic  status. 

■  geographic  iocation/geographic  dis¬ 
persion  '  Distance  learning  courses 
have  been  successful  in  remote  geo¬ 
graphic  locations  with  geographic  dis¬ 
persion  of  the  population  (Schurman  & 
Blackman,  1989). 

■  computer  experience  -  Students  start¬ 
ing  computer-based  courses  have  high 
anxiety  about  the  new  media  and  may 
drop  out  if  they  do  not  begin  with  a 
successful  experience  (Wong  &  Wong, 
1979). 

■  typing  skill  •  Findings  show  that  suc¬ 
cessful  performance  in  computer-me¬ 
diated  training  is  not  contingent  on  the 
typing  ability  of  the  student  (Wells, 
1990). 

■  degree  of  student  autonomy  -  Adult 
learners  tend  to  prefer  autonomous 
learning  situations;  however,  this  may 
not  enhance  course  completion 
(Harbour,  et  aL,  1990). 

■  previous  distance  learning  experience  - 
Findings  show  that  students  who  have 
completed  one  distance  learning  course 
are  very  likely  to  complete  distance 
learning  courses  thereafter  (Rekkedal, 
1983). 

■  self-motivation  -  Findings  show  that  in¬ 
trinsically  motivated  students  internal¬ 
ize  course  material  much  more  than 


their  extrinsically  motivated  counter¬ 
parts  (Marion  &  Svennson,  1982). 

Method  fSubiectsl  Twenty  experts  in  the 
field  of  distance  learning  were  contacted 
in  this  study.  They  represented  four  sec¬ 
tors  (academia,  industry,  government, 
and  military)  and  were  attendees  at  the 
second  Distance  Learning  Conference 
sponsored  by  Los  Alamos  National  Labo¬ 
ratory  in  Scottsdale,  AZ,  in  June  1990. 
The  experts  were  identified  as  those 
having  publications  in  the  field  of  dis¬ 
tance  learning  and  as  those  having  those 
having  ongoing  work  and  influence  in  the 
field. 

Forty-two  percent  of  the  respondents 
were  distance  learning  administrators 
and  more  than  half  of  them  had  more 
than  five  years  of  experience  administer¬ 
ing  distance  learning  courses.  Nineteen 
percent  of  the  respondents  were  course 
designers  and  evaluators  of  distance 
learning  courses.  Only  four  percent  had 
ever  been  students  in  a  distance  learning 
course. 

Their  course  experience  weighed  heavily 
towards  industrial  training.  The  heavy 
training  experience  was  to  be  expected 
because  it  was  sample  dependent;  none 
of  the  conference  attenders  was  involved 
in  distance  learning  at  the  K-12  level. 
There  was  also  a  great  deal  of  course  ex¬ 
perience  in  engineering  (17.5%)  and  sci¬ 
ence  (10%). 

A  wide  variety  of  delivery  systems  was 
used.  Computer-based  training/  com¬ 
puter-aided  instruction  (CBT/CAI)  was 
the  moot  widely  used  (18%),  while  print 
was  the  recond  most  used  (17%).  One 
encouraging  finding  was  that  all  the  me¬ 
dia  that  were  given  as  possible  response 
categories  were  being  used,  even  the  new 
"hi  tech"  media  such  as  audio  graphics, 
video  conferencing,  and  computer  con¬ 
ferencing.  The  respondents  had  diverse 
ejqjerience  with  each  of  the  mediums. 
The  respondents  reported  that  57%  of 
their  courses  were  delivered  by  a  combi¬ 
nation  of  asynchronous  and  synchronous 
methods. 
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Most  of  the  respondents  had  experience 
with  relatively  short  courses.  Twenty-six 
percent  of  the  respondents  were  involved 
in  courses  that  were  less  than  a  college 
quarter  in  length,  while  twenty-two  per¬ 
cent  were  involved  in  semester-long 
courses.  The  respondents  reported  that 
67%  of  the  distance  students  they  had 
dealt  with  were  military  personnel,  20% 
were  graduate  students,  and  15%  were 
undergraduate  students. 

All  types  of  the  suggested  evaluation 
methods  were  being  used  by  the  respon¬ 
dents.  Interestingly  enough,  the  most 
frequent  method  of  evaluation  was  a  "hi 
tech"  method  (computer,  29%),  while  the 
second  most  used  method  was  a  "low 
tech"  method  (pencil/paper,  25%).  All  of 
the  respondents  reported  they  had  per¬ 
formed  some  sort  of  formal  evaluation, 
most  measuring  student  performance, 
cost  effectiveness,  and  student  attrition 
rate.  Sixty-seven  percent  of  the  courses 
were  evaluated  against  some  type  of  a 
control  group  of  which  ninety-two  percent 
were  against  the  traditional  classroom. 

Method  (Survey  Instrument!  The  survey 
<nstrument  was  a  two-part  questionnaire. 
Ibe  first  part  was  the  demographic  data 
of  the  respondents.  It  requested  informa¬ 
tion  about  the  background  and  experi¬ 
ence  the  respondents  had  in  various  as¬ 
pects  of  distance  learning. 

The  second  part  had  the  respondents  rate 
each  student  variable  on  a  Likert  scale  of 
1  to  5  (1  being  not  effective,  5  being  very 
effective).  The  survey  allowed  the  re¬ 
spondents  to  make  comments  throughout 
the  survey.  When  the  respondents  made 
their  comments,  they  were  asked  to  re¬ 
port  if  it  was  an  opinion  or  if  it  was  based 


on  experience.  Many  of  the  responses 
and  comments  were  based  on  actual  ex¬ 
perience. 

Analysis  and  Results  (Analysis  Method- 
ologyl  Because  survey  data  cannot  gen¬ 
erally  be  assumed  to  be  normally  dis¬ 
tributed,  non-parametric  statistical  meth¬ 
ods  were  used  in  this  analysis.  The 
Freidman  Rank  Sum  test  was  used  to 
determine  whether  the  student  variables 
were  perceived  to  differ  with  regard  to 
their  impact  on  the  effectiveness  of  dis¬ 
tance  learning  (relative  to  their  impact  in 
traditional  settings).  The  highly  signifi¬ 
cant  statfsiical  result  (S’  =  48.6,  p  <  0.01) 
indicated  that  the  variables  were,  indeed, 
viewed  as  having  different  impacts  on  dis¬ 
tance  learning  effectiveness. 

Obtaining  this  significant  result  allowed 
us  to  continue  with  our  analysis,  using  the 
Wilcoxon  Rank  Sum  test  to  assess  the 
relative  impact  of  all  possible  pairs  of 
variables.  Basically,  these  analyses  help 
to  answer  the  following  general  question: 
"Is  variable  A  more  or  less  (or  equally) 
important  as  variable  B  in  terms  of  its  in¬ 
fluence  on  distance  learning  effective¬ 
ness?"  These  results  are  detailed  below. 

Finally,  descriptive  statistics  were  used  to 
explore  at  what  levels  within  a  particular 
variable  identified  by  the  Wilcoxon  tests 
as  important  distance  learning  might  be 
most  effective. 

Analysis  and  Results  f  Results!  Systemtic 
patterns  of  results  were  found  for  5  of  the 
12  student  variables  surveyed.  These  are 
each  described  in  turn.  To  help  the 
reader  in  interpreting  these  results, 
means  and  standard  deviations  for  each 
variable  are  shown  in  Table  I. 
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TABLE! 


MEAN  AND  STANDARD  DEVIATION 
OF  EACH  STUDENT  VARIABLE 


Variable 

Mean 

Deviation 

Gender 

2.86 

0.86 

Reading  Level 

4.00 

1.00 

Age 

3.67 

0.98 

Socioeconomic  Status 

3.53 

0.92 

Geographic  Location 

3.62 

0.77 

Education  Level 

3.75 

1.00 

Occupation 

3.30 

0.95 

Computer  Experience 

3.77 

0.72 

Typing  Skill 

3.57 

0.75 

Student  Autonomy 

4.50 

0.73 

Previous  DL  Experience 

3.93 

1.14 

Self-Motivation 

4.73 

0.59 
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■  Gender  *  As  can  be  seen  from  Table  H,  gender.  This  contradicts  evidence  from 

virtually  every  variable  studied  was  the  literature  suggesting  that  gender 

considered  to  be  more  important  to  does  impact  success  in  distance  leam- 

distance  learning  effectiveness  than  ing  courses. 


TABLE  II 

WILCOXON  TESTS 
COMPARING  GENDER 
TO  OTHER  SURVEY  VARIABLES 


Gender  versus. . . 

Variable 

w* 

P 

Reading  Level 

-3.73 

.OOOlt 

Age 

2.74 

.0031 1 

Socioeconomic  Status 

1.85 

.0322t 

Geographic  Dispersion 

2.11 

.0174t 

Education 

2.69 

.0036t 

Occupation 

2.61 

.0045t 

Computer  Experience 

2.75 

.0033 1 

Typing  Skill 

1.92 

.0274t 

Autonomy 

4.05 

.0001 1 

Previous  DL  Experience 

3.07 

.OOllt 

Self-Motivation 

3.38 

.0004t 

Legend 
t  -  significant 
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■  Reading  level  -  Table  ni  shows  that 
reading  level  was  considered  to  be  less 
important  to  distance  learning 
effectiveness  than  only  one  variable 
(namely,  self-motivation).  Reading 
level  was  considered  to  be  more  impor¬ 
tant  than  most  of  the  demographically- 
oriented  variables,  including  socioeco¬ 
nomic  status,  geographic  location, 
computer  experience,  typing  skiil,  and 
occupation. 


Within  the  various  reading  levels,  a  linear 
trend  was  exhibited  with  respondents’ 
viewing  distance  learning  as  more  effec¬ 
tive  as  reading  level  increased;  categories 
rated  with  their  associated  means  and 
standard  deviations  respectively  were  il¬ 
literate  (mean  =  2.27,  3D  =  1.44),  ele¬ 
mentary  (mean  =  2.67,  SD  =  1.18),  ju¬ 
nior  hi^  school  (mean  =  3.07,  SD  =  .83), 
high  school  (mean  =  3.47,  SD  =  .64),  and 
high  school  graduate  (mean  =  4.06,  SD  = 
1.18). 


TABLE  III 

WILCOXON  TESTS 
COMPARING  READING  LEVEL 
TO  OTHER  SURVEY  VARIABLES 


Reading  Level  versus. . . 

Variable 

w* 

P 

Gender 

-3.73 

.0001 1 

Age 

-1.31 

.0951 

Socioeconomic  Status 

-1.67 

.0475t 

Geographic  Dispersion 

-2.19 

.0143 1 

Education 

-1.30 

.0968 

Occupation 

-1.81 

.0351 1 

Computer  Experience 

-1.71 

.0436t 

Typing  skill 

-2.02 

.0217t 

Autonomy 

0.990 

.1611 

Self-Motivation 

2.02 

.0217t 

Previous  DL  Experience 

0.19 

.4247 

l.«gend 
t  =  signincant 
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■  Previous  distance  learning  experience  • 
As  can  be  seen  from  Table  IV,  previous 
distance  learning  experience  was 
viewed  as  more  important  to  future 
success  in  distance  learning  environ¬ 
ments  than  many  of  the  demographi- 
cally-oriented  '.'ariables. 


Respondents  felt  that  distance  learning 
was  much  more  effective  foi  students 
who  had  experienced  a  previous  distance 
learning  course  versus  their  ''nexpe- 
rienced  counterparts  with  means  and 
standard  deviations  of  4.07/1.2,  and 
2.73/1.22,  respectively. 


TABLE  IV 

WILCOXON  TESTS  COMPARING 
PREVIOUS  DISTANCE  LEARNING  EXPERIENCE 
TO  OTHER  SURVEY  VARIABLES 


Previous  Distance  Learning  Experience  versus 

••• 

Variable 

w* 

P 

Age 

-1.14 

.1271 

Socioeconomic  Status 

-1.41 

.0793 1 

Geographic  Dispersion 

-1.71 

.0436t 

Occupation 

-1.86 

.0314t 

Computer  E.xperience 

-1.49 

.0681 

Typing  skill 

-1.63 

.0516t 

Autonomy 

1.10 

.1357 

Self-Motivation 

2.09 

.0183 1 

Legend 
t  =>  significant 


461 


■  Student  autonomy  -  Student  autonomy 
was  also  viewed  as  very  important, 
compared  with  other  student  variables, 
in  ensuring  distance  learning  effective¬ 
ness.  The  result  is  demonstrated  in 
Table  V. 


Respondents  felt  that  distance  learning 
was  very  effective  for  students  with  a  h 
level  of  autonomy  (mean  =4.5,  SD  =.73), 
was  moderately  effective  for  partially  au¬ 
tonomous  students  (mean  =3.53,  SD 
=  .83)  and  had  limited  effectiveness  for 
low  autonomous  students  (mean  =2.46, 
SD  =  1.50). 


TABLE  V 

WILCOXON  TESTS 
COMPARING  AUTONOMY 
TO  OTHER  SURVEY  VARIABLES 


Autonomy  versus. . . 

Variable 

w* 

P 

Age 

-4.05 

.0001 1 

Socioeconomic  Status 

-2.51 

.0060t 

Geographic  Dispersion 

-2.93 

.0017t 

Previous  DL  Experience 

-1  10 

.1357 

Occupation 

-3.11 

.0009t 

Computer  Experience 

-2.57 

.0051 1 

Typing  skill 

-2.96 

.00151 

Self-Motivation 

1.16 

.1230t 

Legend 
t  =  significant 
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■  Self-motivation  -  Finally,  as  can  be 
seen  from  Table  VI,  self-motivation 
was  viewed  as  perhaps  the  most  impor¬ 
tant  student  variable  in  terms  of  con¬ 
tributing  to  distance  learning  effective¬ 
ness. 


Respondents  reported  that  distance 
learning  was  much  more  effective  for 
those  students  with  high  self-motivation 
(mean  =4.66,  SD  =.72)  than  for  students 
with  low  self-motivation  (mean  =2.00,  SD 
=  .84). 


TABLE  VI 

WILCOXON  TESTS  COMPARING 
SELF-MOTIVATION 
TO  OTHER  SURVEY  VARIABLES 


Self-Motivation  versus. .  - 

Variable 

w* 

P 

Age 

-2.02 

.0217t 

Socioeconomic  Status 

-3.21 

.0007t 

Geographic  Dispersion 

-3.66 

.0001 1 

Autonomy 

-1.16 

.1230 

Previous  DL  Experience 

-2.09 

.0183 

Occupation 

-3.81 

.0001 1 

Computer  Experience 

-3.65 

.0001 1 

Typing  Skill 

-3.68 

.0001 1 

Lxgend 
t  =  significant 
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Discussion  Clearly,  student  variables  that 
affect  the  effectiveness  of  distance 
learning  are  very  important.  For  distance 
learning  to  be  a  viable  solution  to  the  ed¬ 
ucation  crisis,  courses  must  be  well  de¬ 
signed  and  students  must  be  carefully  se¬ 
lected.  The  survey  data  have  highlighted 
several  important  student  variables  that 
must  be  considered  in  the  design  of  dis¬ 
tance  learning  courses. 

Reading  level,  student  autonomy,  and 
self-motivation  were  all  identified  as 
having  particular  importance  in  distance 
learning.  Specifically,  students  with  a 
good  reading  ability  and  a  great  deal  of 
autonomy  and  self-motivation  provide  a 
good  target  population  for  distance  learn¬ 
ing.  If  the  instructor  could  select  students 
that  have  these  characteristics,  many  of 
the  learner  discrepancies  in  distance 
learning  would  be  eliminated.  Instead, 
more  often  than  not,  distance  instructors 
do  not  get  to  choose  the  type  of  student 
who  enrolls  for  the  course.  Therefore, 
mechanisms  to  promote  these  important 
characteristics  are  buUt  into  the  course 
design.  Thus,  these  variables  should  be 
assessed  before  a  student  is  assigned  to  a 
distance  learning  course.  Students  not 
possessing  these  qualities  may  require 
special  support.  One  such  "safety  net" 
might  be  to  allow  student-controlled  ac¬ 
cess  to  an  instructor  (via  e-mail,  a  com¬ 
puter  conference,  and/or  a  telephone 
hotline)  so  that  help  is  always  available. 

Most,  if  not  all,  media  in  distance  learn¬ 
ing  promote  a  very  autonomous  learning 
environment  for  students.  Although 
there  is  some  interaction  with  other  stu¬ 
dents  and  the  instructor,  the  majority  of 
the  time  distance  learning  students  work 
independently.  Students  who  need  a  high 
degree  of  structure  and  explicit  instruc¬ 
tions  often  have  a  hard  time  in  the  au¬ 
tonomous  learning  environments  created 
by  distance  learning.  Reading  level 
proved  to  be  an  important  skill  in  dis¬ 
tance  learning  courses  because  most  me¬ 
dia  are  print  media.  There  are  several 
technologies  such  as  audio  graphics, 
CBT/CAI,  and  audio  conferencing  that 
do  not  focus  so  heavily  on  reading.  These 


media  should  be  incorporated  in  course 
design  if  reading  deficiencies  are  ex¬ 
pected.  Thus,  it  is  essential  for  the 
course  design  to  provide  some  type  of 
safety  net  for  non-autonomous  students. 
By  identifying  such  characteristics  at  the 
onset  of  the  course,  the  student,  as  well  as 
the  instructor,  has  a  much  better  possi¬ 
bility  of  succeeding. 

Motivation  is  an  essential  element  in  all 
distance  learning  courses.  Unlike  tradi¬ 
tional  face-to-face  classes  where  students 
are  motivated  by  the  instructor,  distance 
learning  students  are  motivated  by  the 
course  material  and  student  support  ser¬ 
vices.  Baath  (1982)  distinguishes  be¬ 
tween  students  who  are  intrinsically  mo¬ 
tivated  and  extrinsically  motivated.  In¬ 
trinsically  motivated  students  acquire  and 
retain  more  factual  and  conceptual 
knowledge  than  their  extrinsically  moti¬ 
vated  counterparts.  Intrinsically  moti¬ 
vated  students  are  more  likely  to  learn 
concepts  and  logic  while  extrinsically  mo¬ 
tivated  students  will  simply  memorize 
facts.  This  issue  is  another  one  that 
course  designers  and  instructors  need  to 
recognize.  Motivational  tactics  need  to 
be  embedded  in  the  course  material, 
which  can  be  done  several  different  ways. 
Examples  include  giving  short  and  simple 
assignments  at  the  beginning  of  the 
course  to  build  the  students’  confidence 
in  the  distance  learning  course  and  pro¬ 
viding  interactivity  with  the  studen^ 

Conclusion  This  study  has  identified  a 
number  of  student  variables  that  might 
fundamentally  impact  course  design  in 
the  distance  learning  environment.  Fur¬ 
ther  research  is  needed.  As  a  first  step,  a 
more  rigorous  statistical  analysis  will  be 
conducted  on  the  current  data  to  deter¬ 
mine  both  which  variables  are  the  most 
important  and  for  which  levels  of  the 
variables  distance  learning  is  most  effec¬ 
tive  in  a  statistical  sense.  Then,  a  more 
refined  survey  will  be  developed.  In  the 
future,  it  might  be  advantageous  to  take 
this  study  a  step  further  and  gather  in- 
depth  data  on  a  limited  set  of  variables  to 
provide  very  specific  recommendations  to 
course  designers. 
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Distance  learning  can  be  a  viable  solution 
to  the  education  crisis  at  hand.  Distance 
learning  has  the  ability  to  cross  many 
boundaries  that  are  otherwise  closed  to 
education.  We  must  recognize  the  poten¬ 
tial  distance  learning  has  to  enhance  our 


education  system.  At  the  same  time,  we 
must  identify  the  fundamentals  of  dis¬ 
tance  education  so  we  can  begin  to  reap 
the  many  benefits  distance  learning  has 
to  offer. 
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